...making Linux just a little more fun!

May 2010 (#174):


Mailbag

This month's answers created by:

[ Amit Kumar Saha, Ben Okopnik, Henry Grebler, Kapil Hari Paranjape, René Pfeiffer, Mulyadi Santosa, Neil Youngman, Aioanei Rares ]
...and you, our readers!

Our Mailbag


rsync internals

Amit Saha [amitsaha.in at gmail.com]


Thu, 1 Apr 2010 12:54:33 +0530

Hello TAG:

I have been using 'rsync' off late for backups and helping my Prof. do the same as well.

My prof. has ~6G space left on his hard disk and we are trying to back up everything (~200 G)to a external HDD. As 'rsync' progresses, the disk space on the (source) hard disk gets consumed incrementally and large files (~14G) cannot be copied to the backup hard-disk. rsycn fails saying "No disk space left on device". I wonder why. I can try to look into rsync source to see why this happens. But, why is rsync consuming disk space on the source disk? Its a file copy operation right? It should consume the main memory, not the secondary memory. Or, is it a virtual memory thing?

Thanks for any insights!

Best Regards, Amit

-- Journal: http://amitksaha.wordpress.com ?-blog: http://twitter.com/amitsaha

[ Thread continues here (5 messages/6.01kB) ]


compressing sparse file(s) while still maintaining their holes

Mulyadi Santosa [mulyadi.santosa at gmail.com]


Mon, 29 Mar 2010 21:25:23 +0700

Due to certain reasons, we might have sparse file and want to compress it. However, we want to maintain its sparseness. Can we do it the usual way? Let's say we have this file: $ dd if=/dev/zero of=./sparse.img bs=1K seek=400 count=0 0+0 records in 0+0 records out 0 bytes (0 B) copied, 2.3956e-05 s, 0.0 kB/s

$ ls -lsh sparse.img 4.0K -rw-rw-r-- 1 mulyadi mulyadi 400K 2010-03-29 21:14 sparse.img

$ gzip sparse.img $ ls -lsh sparse.img.gz 8.0K -rw-rw-r-- 1 mulyadi mulyadi 443 2010-03-29 21:14 sparse.img.gz

$ gunzip sparse.img.gz $ ls -lsh sparse.img 408K -rw-rw-r-- 1 mulyadi mulyadi 400K 2010-03-29 21:14 sparse.img

Bad. After decompression, total blocks occupied by the file "grows" from 4KiB to 408 KiB.

The trick is by using tar with -S option: $ tar -Sczvf sparse.img.tgz sparse.img

$ ls -lsh sparse.img.tgz 8.0K -rw-rw-r-- 1 mulyadi mulyadi 136 2010-03-29 21:18 sparse.img.tgz

$ tar -xzvf sparse.img.tgz $ ls -lsh sparse.img 4.0K -rw-rw-r-- 1 mulyadi mulyadi 400K 2010-03-29 21:17 sparse.img

As you can see, the total block size of "sparse.img" are correctly restored after decompression.

-- 
regards,
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com

[ Thread continues here (3 messages/3.53kB) ]


C/C++ autocompletion for Emacs/vim

Jimmy O'Regan [joregan at gmail.com]


Sat, 24 Apr 2010 12:04:21 +0100

http://cx4a.org/software/gccsense/

-- <Leftmost> jimregan, that's because deep inside you, you are evil. <Leftmost> Also not-so-deep inside you.

[ Thread continues here (6 messages/3.63kB) ]


Resuscitating a comatose xterm window?

Kat Tanaka Okopnik [kat at linuxgazette.net]


Sun, 18 Apr 2010 20:15:57 -0700

Hi, gang -

After a history of losing data to dying computers (yes, yes, I know, it's really about not backing up data adequately), I've come to prefer leaving my e-mail on remote servers rather than dealing with it locally. As a result, my non-webmail interaction is done via ssh. I've learned to run screen so that a dropped connection doesn't result in losing half-composed mail, but I haven't figured out what to do with my frozen xterm window.

What I mean by this is that I end up losing the connection, having to reconnect, and then having to kill the frozen-by-dropped-connection window. Is there a way that I can resuscitate the dead session in the existing window, rather than opening up a new xterm window and killing the old one?

Please be gentle as I expose my lack of proper terminology. I'm pretty sure I'm not describing my problem using all the right words, but I'm hoping I've come close enough for someone to figure out how I should have put it. (I'd ask my resident *nix guru to vet my question, but he's up against a deadline while running short on sleep.) I'd wait to ask my question, but I suspect I'd forget to ask, again...

-- Kat Tanaka Okopnik Linux Gazette Mailbag Editor kat at linuxgazette.net

[ Thread continues here (18 messages/30.46kB) ]


how to close connected/listening socket?

Mulyadi Santosa [mulyadi.santosa at gmail.com]


Sun, 18 Apr 2010 15:38:13 +0700

Hi all

Once I read there is a way to close a listening/half open/connected/etc TCP socket using Linux kernel facility i.e using one of /proc entries.

Do any of you recall which /proc (or /sys maybe?) entry that does the job?

Thanks in advance..

-- 
regards,
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com

[ Thread continues here (3 messages/2.31kB) ]


LCA: How to destroy your community

Jimmy O'Regan [joregan at gmail.com]


Tue, 20 Apr 2010 23:54:03 +0100

http://lwn.net/Articles/370157/

Great article. A co-contributor and I have been picking through it point by point to see how many points our project is guilty of (mostly documentation, or the lack thereof :)

-- <Leftmost> jimregan, that's because deep inside you, you are evil. <Leftmost> Also not-so-deep inside you.

[ Thread continues here (2 messages/1.49kB) ]


how to unmount stale NFS connection

Mulyadi Santosa [mulyadi.santosa at gmail.com]


Mon, 19 Apr 2010 01:05:38 +0700

Hi all

Following my recent question regarding how to close tcp connection, it turns out that in my case, the sockets belong to stale NFS connection. In other words, the client try to maintain a mount toward a server that no longer exist.

So, how to unmount it? If you use the usual umount or umount.nfs, you can't do it and it will yield error messages like below: $ sudo umount.nfs /media/nfs -f -v umount.nfs: Server failed to unmount '192.168.1.2:/opt/data' umount2: Device or resource busy umount.nfs: /media/nfs: device is busy

Thanks to Google, I came to this URL http://www.linuxquestions.org/linux/answ[...]_gone_down_causing_the_applications_thre (hopefully I paste the entire URL correctly).

Essentially, the trick is to "emulate" the dead NFS server. Bring up a virtual interface such as eth0:0 and assign it the IP address of the dead server: # ifconfig eth0:0 192.168.1.2 netmask 255.255.255.0 # ifconfig et0:0 up

Then retry the umount or umount.nfs command. It takes times, but eventually it will succeed. Finally, bring down the virtual interface: # ifconfig eth0:0 down

PS: Big credits to Rahul K. who share this valuable information.

-- 
regards,
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com

[ Thread continues here (3 messages/3.28kB) ]


Remote mail usage (Was Resuscitating a comatose xterm window?)

Kapil Hari Paranjape [kapil at imsc.res.in]


Tue, 20 Apr 2010 10:05:28 +0530

Hello,

In the context of remote access to mail, On Mon, 19 Apr 2010, Kat Tanaka Okopnik wrote:

> I tried running a screen session locally, and then ssh to where my
> mail lives, but that resulted in some keystrokes getting interpreted in
> ways that I didn't like (e.g. the backspace key becomes something else,
> IOW not backspacing). This is probably a whole different question that
> should go under another subject line...

So I thought I should try to put together some thoughts on remote usage of mail. Since this has nothing to do with the original question, I have broken the thread.

Case 1: Remote system only allows webmail access
------------------------------------------------
This is unfortunately all too common nowadays. Possible solutions:

 1. Live with it/abandon it entirely
 2. If forwarding is allowed, then forward your mail to a "better"
 server.
 3. "freepops" may work for you. This is a program that acts as a
 POP3 front-end to a number of webmail interfaces.
Case 2: Remote system only allows IMAP/POP access
-------------------------------------------------
This is slightly more common, yet often sub-optimal!
 1. Setup your favourite mail client to access imap/pop.
 2. Use one of the many programs that fetches mail from such servers.
Each of these has its own problems. In the first case, you do not get offline access to your mail, while in the second case you might end up paying a lot to download an attachment that you did not really need to access right away.

Case 3: Remote system allows shell access
-----------------------------------------
This is the best case scenario that is increasingly scarce!
 1. Login to system and use whatever client is locally available. In
 addition you may want to run this client under "screen" or "dtach"
 in order to re-use sessions in case of lost connections.
 
 2. Filter the mail on the remote system (using procmail and friends)
 to sort out the mail (optionally separating out attachments). Then
 download the mail you really want to read using a number of different
 techniques (rsync, unison, git and others or those mentioned in Case 2).
Clearly, Kat was using (Case 3, Para 1).

Kapil. --

[ Thread continues here (2 messages/5.71kB) ]


VirtualBox Tricks?

clarjon1 [clarjon1 at gmail.com]


Thu, 25 Mar 2010 09:20:19 -0400

Hey Gang! I don't know how many of you use VirtualBox (OSE or otherwise), but I was wondering if any of you had any cool tricks-of-the trade that perhaps readers might be able to use. To start off with, here's one that makes VBox really easy for my mom to use, a little something I've called the "XP Start Button": It's essentially an icon on the desktop/menu entry. I've upgraded her laptop to find that she's never ever had a need to boot up windows, but unfortunately the tax guy wants her to run windows specific software. Anyways, here's the tip: Create a .desktop file in a folder/desktop, and fill it with the following:

[Desktop Entry]
Encoding=UTF-8
Version=1.0
Type=Application
Terminal=false
Name[en_US]=Windows XP
Exec=VirtualBox --startvm "Windows XP"
Name=Windows XP

Of course, replacing the name of the virtual machine with the one you wish to start on opening of the .desktop file.

Hope to hear more tips, wonder what tweaks i might learn :)


Sync error: ToC on Homepage and current issue page

Amit Saha [amitsaha.in at gmail.com]


Fri, 2 Apr 2010 10:36:14 +0530

Hello:

The contents for April on the homepage and on the issue page seem to be out of sync: http://linuxgazette.net/173/index.html

Hope, I am not seeing this in the middle of a update.

-Amit

-- Journal: http://amitksaha.wordpress.com ?-blog: http://twitter.com/amitsaha

[ Thread continues here (5 messages/8.42kB) ]



Talkback: Discuss this article with The Answer Gang

Published in Issue 174 of Linux Gazette, May 2010

Talkback

Talkback:173/starks.html

[ In reference to "Picking Fruit" in LG#173 ]

Henry Grebler [henrygrebler at optusnet.com.au]


Mon, 05 Apr 2010 11:53:28 +1000

Hi Ken,

Great article. And you sound like a beautiful person.

But enough of this man love.

Re melons. Someone of the medical persuasion taught me that the way to pick melons is to percuss them (see http://en.wikipedia.org/wiki/Percussion_%28medicine%29). I am not myself medically inclined, but I have applied this process for several years. To use your metaphor, I certainly do not bat 1.000. I couldn't tell you for sure how well I bat, but I'd guess up around .7 (I won't dignify it with 3 decimal places since it could be .6). But I am quietly confident that I bat much better these days than I used to.

I never bring home concrete blocks; the ones I bring home taste from ok to excellent.

On the subject of HeliOS, I probably could not be further from Central Texas. Your project captures my imagination, but at this distance I can only offer moral support.

I have some ideas related to picking candidates for Linux conversion, based on points raised in your article. I realise that I do not have a complete understanding of the environment in which you find yourself, but I respectfully offer these as possibly useful, perhaps with some adaptation.

We all have an investment in what we know. You said it yourself: "People get used to doing things one way." I'm writing this on a new platform (second-hand hardware). I got fed up with Fedora when I discovered that F10 needed a minimum of 512MB of RAM. I have a Redhat 8 firewall which until recently was running just fine on a 24MB Pentium 1 100. So I've installed FreeBSD. My /History file indicates that I started installing 17 January 2010. While I was building I still had my old Fedora machine. Eventually (16 March 2010), I decided to bite the bullet and move to FreeBSD.

But I'm stilling running my old Fedora machine. I've got at least a dozen xterm sessions SSHed into it. I keep discovering things I want to be able to do, things I can do on the Fedora box that I can't yet do on my FreeBSD box. Gradually I am installing the software I need.

And, boy, am I glad that I had the safety net. Recently, I rebooted my FreeBSD machine because I'd installed quite a lot of software and I wanted to be sure it would still work after a reboot - and it didn't! In the event, it was only off the air for a day. But I know that from the perspective of today looking backwards. Faced with a machine hung somewhere in the boot process, my first reaction was panic. And terror.

Further, without the ability to fallback to my other machine, I would not have been able to get onto the forums which eventually gave me a pointer about how to work around my problem.

Perhaps some of the prospects for conversion to Linux need the safety net of being able to go back to the familiar environment of Windows until they can be weaned off.

[ ... ]

[ Thread continues here (1 message/6.01kB) ]


Talkback:171/nonstyleguide.html

[ In reference to "Words, Words, Words" in LG#171 ]

Henry Grebler [henrygrebler at optusnet.com.au]


Tue, 06 Apr 2010 10:32:43 +1000

Hi Gang,

In his article, "Words, Words, Words", Rick complained about

"Levitz is having a sale at their Oakland warehouse."

I was going to write explaining Rick's objection. But that would be naughty. Instead, I'll explain mine.

The sentence uses number inconsistently.

I don't care if "Levitz" is singular or plural - so long as it is only one, not both - especially within a single sentence. It defies logic for an entity to be simultaneously singular and plural.

So, I'm ok with either of these:

"Levitz is having a sale at its Oakland warehouse." "Levitz are having a sale at their Oakland warehouse."

Cheers, Henry



Talkback: Discuss this article with The Answer Gang

Published in Issue 174 of Linux Gazette, May 2010

2-Cent Tips

[2 cent tips] Better reboot using magic SysRq

Mulyadi Santosa [mulyadi.santosa at gmail.com]


Tue, 27 Apr 2010 07:26:47 +0700

What is a better way to reboot your system in case it hangs? I mean rather than three finger salute or kicking the power switch? :)

Let's assume you have enable the magic SysRq before. Make sure this feature is compiled in the kernel (CONFIG_MAGIC_SYSRQ) and you enable it (echo 1 > /proc/sys/kernel/sysrq). Then when the disaster strike, you can try pressing these key combos in the following order: 1. Alt-SyRq-s (sync the buffer with the disk) 2. Alt-SysRq-u (remount all filesystem as read-only) 3. Alt-SysRq-e (kill all tasks, except init) 4. Alt-SysRq-b (finally, reboot system)

If your keyboard doesn't have SysRq, use PrintScreen instead. The way I do it is like this: press alt, hold, press sysrq, hold, then press s (or other key), then release them all. You might also try, press alt, hold, press sysrq, release it, press s or other key, release all.

In most cases, the above sequence could reboot your system better. But if not, just fall back to the "traditional" one :)

Reference: sysrq.txt in Documentation directory inside kernel source code's tree.

-- 
regards,
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com

[ Thread continues here (5 messages/3.52kB) ]


2-cent Tip: Screen Shots

Ben Okopnik [ben at linuxgazette.net]


Tue, 13 Apr 2010 15:56:07 -0400

----- Forwarded message from Henry Grebler <henrygrebler at optusnet.com.au> -----

I was writing an article. I needed more than words; I needed screen shots. Over the years, I've done this in different ways. Of late, I've used ImageMagick(1) for this sort of work, in particular,

import - saves any visible window on an X server and outputs it as an image file. You can capture a single window, the entire screen, or any rectangular portion of the screen.

I was capturing parts of web pages. Then I came to a roadblock. I clicked on a button and it produced a drop-down list. I wanted to capture the contents of the drop-down list. But, some drop-down lists are unfriendly. For example, in Firefox, if I click on File, the usual drop-down list appears, but now my keyboard is deactivated. I also lose most mouse functions, except for moving the cursor. And, if I click anywhere, the drop-down list disappears.

So how to take the screen shot?

I started up a vncserver and a vncviewer (client); and then invoked Firefox inside the VNC client.

Click on File, move the cursor out of the VNC client, and, voila - I can now use import to take a screen shot of the menu of the Firefox running in vncviewer.

----- End forwarded message -----

[ Thread continues here (3 messages/5.74kB) ]



Talkback: Discuss this article with The Answer Gang

Published in Issue 174 of Linux Gazette, May 2010

News Bytes

By Deividson Luiz Okopnik and Howard Dyckoff

News Bytes

Contents:

Selected and Edited by Deividson Okopnik

Please submit your News Bytes items in plain text; other formats may be rejected without reading. [You have been warned!] A one- or two-paragraph summary plus a URL has a much higher chance of being published than an entire press release. Submit items to bytes@linuxgazette.net. Deividson can also be reached via twitter.


News in General

lightning boltUtah Jury Confirms Novell Owns Unix Copyrights

At the end of March, the jury for the District Court of Utah trial between SCO Group and Novell issued a verdict determining the ownership of important Unix copyrights.

The jury's decision confirmed Novell's ownership of the Unix copyrights, which SCO had asserted to own in its attack on Linux. Novell stated that it remains committed to promoting Linux, including by defending Linux on the intellectual property front.

Ron Hovsepian, Novell president and CEO, said "This decision is good news for Novell, for Linux, and for the open source community. We have long contended that this effort against Linux has no foundation, and we are pleased that the jury, in a unanimous decision, agrees. I am proud of Novell's role in protecting the best interests of Linux and the open source community."

SCO is reported as currently weighing an appeal and it may also continue its related lawsuit against IBM for both Unix infringement and some contract conflicts.

There is a little more on the tail end of this saga. From http://www.groklaw.com:

"The SCO bankruptcy hearing on the sale of the Java patent will be on April 20. That's this proposed sale to Liberty Lane for $100,000, and that's an LLC affiliated with Allied Security Trust, the anti-patent-trolls company, if you've dropped a stitch and can't keep up as SCO's assets get sold off bit by bit. If anyone else has bid, other than Liberty Lane, then there would be an auction on the 19th, but SCO told the court they don't expect that to happen."

lightning boltHP ranked 10th in FORTUNE 500 List

HP has entered the coveted top 10 category of the Fortune 500 list.

IBM came in at 20th, Microsoft at 36th, Dell at 38th, and Apple 56th, just ahead of Cisco at 58th. Amazon.com was 100th with $24 B in revenue. Oracle was 105th at $23 B, but Sun was separately listed at 204th with 11 B in revenue. Combined with Sun's revenue, the new Oracle would be competing with Apple and Cisco for their rankings in the mid-50s of the list.

See the full list here: http://money.cnn.com/magazines/fortune/fortune500/2010/.

lightning boltMySQL Advisory Board Meetings set for May

Oracle has announced that the MySQL Customer Advisory Board meetings will take place on May 6, 2010 in Amsterdam and June 3, 2010 in Redwood Shores, Calif. The meetings will serve as a forum for MySQL customers to learn about new enhancements to the MySQL database, and provide feedback. The Storage Engine Advisory Board will focus on MySQL's storage engine API, and its future.

The announcement coincided with the recent MySQL User Conference which will remain separate from the annual Oracle OpenWorld, currently being held in San Francisco this September.

Also, Oracle Announced MySQL Cluster 7.1, which introduces MySQL Cluster Manager, a new solution that simplifies and automates the management of the MySQL Cluster database. More details are in the Products section of News Bytes, below.

lightning boltUbuntu 10.04 Long Term Support (LTS) Server Reaches for the Clouds

Canonical has released Ubuntu 10.04 LTS Server Edition with extended security and maintenance updates to all users for five years. Ubuntu 10.04 LTS extends the cloud-computing capability of Ubuntu Enterprise Cloud with Eucalyptus, an open source cloud management tool for private computing clouds.

More info in the Distro section of News Bytes, below.

lightning boltSynaptics' Advanced Gesture UI Moves To Linux Operating Systems

In April, Synaptics announced the extension of its industry-leading Synaptics Gesture Suite to the Linux operating system environment. This release extends the Synaptics Gesture Suite - which includes sophisticated multi-finger gestures - to OEMs that offer Linux-based solutions.

Synaptics has a strong leadership position in the Microsoft Windows-based TouchPad market and Synaptics Gesture Suite (SGS). Today's announcement extends the industry's broadest gesture suite across a wide range of leading Linux operating systems.

"The Synaptics Gesture Suite for Linux enables our OEMs to leverage a broad range of gesture capabilities across Linux operating systems, and offers extensibility into new Linux flavors such as Google Chrome OS and additional support for touch-enabled remote control devices," said Ted Theocheung, head of Synaptics PC and digital home products ecosystem. "SGS ensures optimized interoperability of gestures, minimal gesture interpretation errors, and proven usability performance across the widest range of TouchPad sizes."

Supported Linux operating systems include Fedora, Millos Linpus, Red Flag, SLED 11 (SuSE), Ubuntu, and Xandros. SGS for Linux (SGS-L) supports a wide range of pointing enhancements and gestures including two-finger scrolling, PinchZoom, TwistRotate, PivotRotate, three-finger flick, three-finger press, Momentum, and ChiralScrolling. Bundled with Synaptics' enhanced driver interface, SGS-L is provided free of charge to Synaptics OEM/ODM partners when ordered with Synaptics TouchPad and ClickPad products.

Synaptics Gesture Suite for Linux (SGS-L) helps manufacturers bring new interactivity and productivity to their notebook PC systems and other peripheral devices that use Synaptics TouchPads. To find out more about Synaptics Gesture Suite for Linux, please visit: http://www.synaptics.com/go/SGSL.

lightning boltGnome and KDE will co-host 2011 Dev Conference

Following the successful Gran Canaria Desktop Summit in 2009, the GNOME Foundation and KDE e.V. Boards have decided to co-locate their flagship conferences once again in 2011, and are taking bids to host the combined event. Desktop Summit 2011 plans to be the largest free desktop software event ever.

In July of 2009, the GNOME and KDE communities came together for the Gran Canaria Desktop Summit, the first co-located KDE/GNOME event. It was a major success and an opportunity for the leaders of the free software desktop efforts to communicate on common issues and attend combined social events. The attendees from both projects expressed great interest in repeating the event and merging the programs to synchronize schedules and make the event an even greater opportunity for the KDE and GNOME teams to learn from each other and work together.

"The Gran Canaria Desktop Summit was a great first event," said Vincent Untz, GNOME Foundation Board Member. "We enjoyed working with our KDE friends at GCDS in 2009, and want to increase our cooperation in 2011. We plan to go beyond simple co-location this time, and actually plan a combined schedule in 2011 so that KDE and GNOME contributors have every opportunity to work with and learn from each other."

The GNOME and KDE projects will hold independent events in 2010. GUADEC, the GNOME Project's annual conference, will be held in The Hague, Netherlands on July 24 through July 30 of this year. KDE's Akademy will be located in Tampere, Finland from July 3 to 10 this year. Both groups will likely hold smaller sprints through 2010 and early 2011 to prepare for the combined 2011 Desktop Summit.

"The KDE e.V. board felt that GCDS was a fantastic event, and we learned what works well and what can be improved when co-hosting an event with our GNOME friends," said Cornelius Schumacher of the KDE e.V.

More than 850 contributors to the GNOME and KDE projects gathered in Gran Canaria last July. The event brought together attendees from 50 countries, and helped raise local awareness of free software and had a measurable impact on the local community. The impact of the event continues with nearly 2 million hits to the summit Web site following the event.

The projects are seeking a host in Europe at a location that can handle more than 1,000 participants. For detailed requirements, prospective hosts can see the requirements for Akademy (http://ev.kde.org/akademy/requirements.php) and GUADEC (http://live.gnome.org/GuadecPlanningHowTo/CheckList). Applications are welcome before May 15th, and should be sent to the KDE e.V. (kde-ev-board@kde.org) and the GNOME Foundation (board@gnome.org) boards.


Conferences and Events

IBM Impact 2010 Conference
May 2-7, Las Vegas, Nev.
IDN Promotion Code: IMP10IDN
http://www-01.ibm.com/software/websphere/events/impact/index.html.
Citrix Synergy-SF
May 12-14, 2010. San Francisco, Ca
http://www.citrix.com/lang/English/events.asp.
Apache Lucene EuroCon 2010
May 18-21, Prague, Czech Republic
http://lucene-eurocon.org/.
Google IO Developer Conference - now closed
May 19 - 20, Moscone Center, San Francisco, CA
http://code.google.com/events/io/2010/.
Black Hat Abu Dhabi 2010
May 30 - Jun 2, Abu Dhabi, United Arab Emirates
http://www.blackhat.com/html/bh-ad-10/bh-ad-10-home.html.
Internet Week New York
June 7-14, 2010, New York, New York
http://internetweekny.com/.
Southeast Linuxfest (SELF 2010)
June 12-13, Spartanburg, SC

The Southeast LinuxFest is a community event for anyone who wants to learn more about Linux and Free & Open Source software. It is part educational conference, and part social gathering. Like Linux itself, it is shared with attendees of all skill levels to communicate tips and ideas, and to benefit all who use Linux/Free and Open Source Software. The Southeast LinuxFest is the place to learn, to make new friends, to network with new business partners, and most importantly, to have fun!

Register today (FREE to attend)! Book your hotel room now and save!
http://www.southeastlinuxfest.org/registration/form

SELF 2010
http://www.southeastlinuxfest.org
Über Conf 2010
June 14-17, Denver, CO
http://uberconf.com/conference/denver/2010/06/home.
Semantic Technology Conference
June 21-25, Hilton Union Sq., San Francisco, CA
http://www.semantic-conference.com/.
O'Reilly Velocity Conference
June 22-24, 2010, Santa Clara, CA
http://en.oreilly.com/velocity.
USENIX Conference on Web Application Development (WebApps '10)
June 23-24, Boston, MA

Join us for the USENIX Conference on Web Application Development (WebApps '10) which will take place June 23-24, 2010, in Boston, MA, as part of the USENIX Federated Conferences Week.

WebApps '10 is a new technical conference filled with the latest research in all aspects of developing and deploying Web applications. The diverse 2-day conference program will include invited talks by industry leaders including Adam de Boor, Google, on "Gmail: Past, Present, and Future," a variety of topics and new techniques in the paper presentations, WiPs, and a poster session. Since WebApps '10 is part of USENIX Federated Conferences Week, you'll also have increased opportunities for interaction and synergy across multiple disciplines.

Register by June 1 and save up to $400! Additional discounts are available.
http://www.usenix.org/webapps10/lg

WebApps '10
http://www.usenix.org/webapps10/lg
2010 USENIX Annual Technical Conference (USENIX ATC '10)
June 23-25, Boston, MA

Join us for the 2010 USENIX Annual Technical Conference (USENIX ATC '10) which will take place June 23-25, 2010, in Boston, MA, as part of the USENIX Federated Conferences Week.

Don't miss the 3-day conference program filled with the latest innovative work in systems research, including invited talks by Ivan Sutherland on "Some Thoughts About Concurrency," Ben Fry on "Visualizing Data," and Matt Welsh on "RoboBees." Find out about the latest in groundbreaking systems research in the paper presentations, WiPs, and a poster session. Since USENIX ATC '10 is part of USENIX Federated Conferences Week, you'll also have increased opportunities to network with peers across multiple disciplines.

Register by June 1 and save up to $400! Additional discounts are available.
http://www.usenix.org/atc10/lg

ATC '10
http://www.usenix.org/atc10/lg
CiscoLive! 2010
June 27-July 1, Mandalay Bay, Las Vegas, NV
http://www.ciscolive.com/attendees/activities/.
GUADEC 2010
July 24-30, 2010, The Hague, Netherlands
http://www.guadec.org/.
Black Hat USA
July 24-27, Caesars Palace, Las Vegas, Nev.
http://www.blackhat.com/html/bh-us-10/bh-us-10-home.html.
LinuxCon 2010
August 10-12, 2010, Renaissance Waterfront, Boston, MA
http://events.linuxfoundation.org/events/linuxcon.
USENIX Security '10
August 11-13, Washington, DC
http://usenix.com/events/sec10.
LinuxCon Japan 2010
September 27-29, Roppongi Academy, Tokyo, Japan
http://events.linuxfoundation.org/events/linuxcon-japan/.


Distro News

lightning boltUbuntu 10.04 Long Term Support (LTS) Server Reaches for the Clouds

Canonical has released Ubuntu 10.04 LTS Server Edition with extended security and maintenance updates free of charge to all users for five years (versus 18 months for a standard release). The third and latest LTS version of the popular Linux desktop distribution, Ubuntu 10.04 LTS delivers on the standing commitment to release a version of Ubuntu every two years and builds on the success of Ubuntu 8.04 LTS. It has been available for free download since April 29th.

Ubuntu 10.04 extends the cloud-computing capability of Ubuntu Enterprise Cloud with Eucalyptus - a technology that is becoming widely used as a basis for building private clouds. Ubuntu 10.04 LTS also includes 20 major packages added or updated since the previous LTS release, giving new and upgrading users a wide range of applications that can be easily installed at launch.

Over 75 organizations have signaled their intent to certify applications on the platform including Alfresco, Ingres, IBM, VMware, Zimbra, Yahoo and many others with more expected to follow post-launch. Dell has announced its intention to support Ubuntu 10.04 LTS Server Edition and will offer Ubuntu Enterprise Cloud as an option on their PowerEdge-C product line - servers specifically designed for building cloud environments. Intel has also been assessing Ubuntu Enterprise Cloud and jointly published a white-paper with Canonical on how to best deploy the technology on Intel's latest Xeon processors.

New features in Ubuntu 10.04 LTS Server Edition:

* Direct upgrade path for Ubuntu 8.04 LTS and Ubuntu 9.10 users;

* Stability and security enhancements for LTS including:
-- Five years of security and maintenance updates free to all users;
-- AppArmor security by default on key packages;
-- Kernel hardening (memory protection, module loading blocking, address space layout randomization);
-- Encrypted Home and Private directories;

* Virtualization enhancements including:
-- KVM now support KSM memory aggregation and live migration of virtual machines;
-- Automated and fast image creations with VMBuilder;
-- Ubuntu as a Virtual Machine (VM) now supported for VMWare, Xen, KVM, Virtualbox, EC2 and UEC;

* Ubuntu Enterprise Cloud (UEC) and Amazon EC2 enhancements including:
-- EC2 and UEC images are included in five years of free security and support updates;
-- Minimal installation profile for minimum footprint VMs optimized for EC2;
-- Ubuntu EC2 images can be booted from EBS ;
-- Easily customized images at boot time for super flexibility using cloud-init;
-- Puppet configuration management framework can be used to mass control instances from their start.

* Advanced storage capabilities, including Raid and iSCSI and multipath support;

* Simpler to mass deploy and manage:
-- Puppet integration from the installer and in the cloud.
-- Version control changes (integrated with Puppet) provides history and accountability.
-- Many new and improved installation profiles.

Linux Gazette spoke with Nick Barcet, Ubuntu Server Product Manager, and Gerry Carr, Platform Marketing Manager, just before 10.4 was released.

Carr said that this was the third release of Ubuntu Enterprise Cloud (UEC) and first on an LTS release. "This is a significant upgrade," Carr told LG. "This is well-positioned to take us toward more mission-critical deployments in the data center."

"This release will also show a significant shift in ISP support," Carr said, "We expect about 75 ISVs will declare support for 10.04 Server when released and more will follow. ...This is more than doubling the number of ISVs supporting Ubuntu."

Barcet explained that many IT depts are experimenting with Cloud architectures and seem to considering UEC in hybrid public and private architectures. There are now about 200 UEC deployments a day and these are approaching 11,000 total deployments, he said.

Ubuntu 10.10 is scheduled for release in October and Ubuntu 11 is scheduled for the Spring of 2011. The next LTS release will be in 2012.

Ubuntu 10.04 LTS Server Edition, including Ubuntu Enterprise Cloud, is widely available for free download from http://www.ubuntu.com Users can upgrade directly to Ubuntu 10.04 LTS from the previous Ubuntu 8.04 LTS version as well as the Ubuntu 9.10 release.

lightning boltUbuntu 10.04 LTS Desktop has a new look and social network ties

The new desktop edition of Ubuntu 10.04 LTS will feature extensive design work, a nimbler UI, faster boot speed, social network integration, online services and the Ubuntu One Music Store.

Ubuntu 10.04 LTS Desktop also includes three years of support through free security and maintenance updates.

"Ubuntu 10.04 LTS challenges the perceptions of the Linux desktop, bringing a whole new category of users to the world of Ubuntu," said Jane Silber, CEO, Canonical. "Changes like the new look and feel and the addition of a music store, layered on top of our relentless focus on delivering an intuitive and attractive user experience for new and existing Ubuntu users -- these are the bridging elements to the mainstream market that our community, our partners and our users really want. Long-term support makes Ubuntu 10.04 LTS very attractive to corporate IT as well."

New in 10.04 LTS:

* Boot speed: Noticeably quicker on almost any machine.
* Social from the start: The new 'Me Menu' in Ubuntu 10.04 LTS consolidates the process of accessing and updating social networks including Facebook, Digg, Twitter and Identi.ca. The Me Menu also integrates chat channels.
* Ubuntu One: Enhanced desktop integration for the online service means files and folders and bookmarks can be shared and saved on the cloud more easily.
* Ubuntu One Music Store: Music from the world's largest labels and greatest bands available direct to Ubuntu users through the default music player. Purchase tracks, store in Ubuntu One and share DRM-free music from one location across multiple computers and devices.
* Ubuntu Software Centre 2.0: An easy way to find new software, and keep track of it once it's installed.
* Ubuntu 10.04 Netbook Edition (UNE): As well as benefiting from the improvements in the Desktop Edition, netbook users will see even faster boot speeds on SSD-based devices, and faster suspend/resume that will extend battery life.

Ubuntu 10.04 LTS Desktop Edition and Ubuntu 10.04 Netbook Edition are entirely free of charge and are available for download from http://www.ubuntu.com.

lightning boltFedora 13 "Goddard" Beta aims to be a rocket

The Beta release of Fedora 13 (codename "Goddard") blasted off mid-April, named for scientist and liquid-fuel rocketry pioneer Robert Goddard. Only critical bug fixes will be pushed as updates for the general release of Fedora 13, scheduled to be released in the middle of May.

The available Live images make it easy to try out Fedora - if you write one to a USB key, you can add personal data and your favorite applications as you go.

Here are some of the changes that are propelling the Fedora 13 Beta:

* The goal in Goddard is to automate some of the hardware and software tasks that help users get their work done. When the user plugs in a USB or parallel port printer, inserts a specialized CD such as collections of music files as opposed to a standard audio CD, or downloads or opens an archive file, using powerful back end desktop technologies, PackageKit is designed to detect the user's action and offer to install software helpers.

* The Fedora 13 Beta allows developers to install and try a parallel-installable Python 3 stack for the first time. This feature allows developers to write and test code using either the current Python 2.6 or the next-generation Python 3 language.

* There are new functions in Fedora 13 Beta for the GNU debugger, gdb, that allow it to deliver unified information for C/C++ libraries and Python in the same running process. Programmers who are writing Python code that wraps or calls C/C++ functions to enhance performance and rapid development can now more quickly and efficiently detect and debug problems in their code using this work.

* System administrators can also try out some of the advanced file system improvements in Fedora 13 Beta. New features in the btrfs file system allow for rollbacks of entire file system states, making application testing and system recovery more powerful than ever.

* During its development cycle, Fedora 13 also featured for the first time an installable version of Zarafa, a drop-in groupware replacement for Exchange with full featured email, calendaring, and other collaboration tools for use by both Linux and Microsoft clients. A usable and familiar Web interface for users, and support for POP/IMAP and other protocols, is included along with tools for integration with existing Linux services.

To try Fedora 13 Beta in both 32-bit and 64-bit versions, visit http://fedoraproject.org/get-prerelease.

lightning boltMepis antiX 8.5 Released

April saw the release of antiX MEPIS 8.5, a lightweight MEPIS variant designed for older computers. Nine months after the release of antiX M8.2, antiX MEPIS 8.5 - a fast, light, flexible and complete desktop and live CD based on SimplyMEPIS and Debian 'testing' - is now available in full and base versions.

This release defaults to a fully customised IceWM desktop (Fluxbox, wmii and dwm are also installed) using a SimplyMEPIS 2.6.32 kernel, tweaked MEPIS Assistants for better compatibility in antiX and a range of applications for desktop use, including Iceape for Internet needs, AbiWord and Gnumeric for office use, XMMS and Google's music manager for audio, gxine, MPlayer and GNOME MPlayer for video.

This antiX release works on computers with as little as 64 MB RAM, though 128MB RAM is the recommended minimum, and the 486 kernel version should work on AMD K5/K6.

Download antiX 8.5 here: http://ftp.cc.uoc.gr/mirrors/linux/mepis/released/antix/.


Software and Product News

lightning boltXen.org Releases Xen 4.0 Hypervisor for the Cloud

Xen.org, the home of the open source Xen hypervisor, has released Xen 4.0, a collective effort of a global development team representing more than 50 leading technology vendors, universities, and virtualization experts. Leveraging the latest network cards optimized for virtualization, Xen 4.0 provides users substantial performance and scalability gains for any level of enterprise or cloud application workload.

Xen 4.0 adds significant memory and security optimizations that will drive virtualization and infrastructure performance. As a result, virtualization is made suitable for all workloads, even network intensive and high performance computing applications that would have previously experienced compromised performance on any hypervisor. Xen 4.0 enables virtualization to be deployed ubiquitously, across every server in a datacenter, bringing ease of management, secure architecture, high availability, agility and efficiency to all applications.

Key Highlights in Xen 4.0:
- Fault Tolerance - Xen 4.0 now supports live transactional synchronization of VM states between physical servers as a basic component, enabling administrators to guarantee a high degree of service reliability.
- High Availability - Xen 4.0 leverages advanced reliability, availability and serviceability (RAS) features in new Intel Xeon (Nehalem-EX) and AMD Opteron processors.
- Netchannel2 - Xen 4.0's NetChannel2 takes advantage of significant advancements in networking hardware such as SMART NICs with multi-queue and SR-IOV functionality.
- Blktap2 - A new virtual hard disk (VHD) implementation in Xen 4.0 delivers high performance VM snapshots and cloning features as well as the ability to do live virtual disk snapshots without stopping a VM process.
- PVOps Domain 0 - Xen 4.0 is the first release from Xen.org to formally support PVOps in the Domain0 (Dom0) Linux kernel; this new kernel option allows administrators access to the most recent devices supported by the Linux kernel.
- Memory Enhancements - New algorithms, such as Transcendent Memory and Page Sharing have been introduced in Xen 4.0 to enhance the performance and capabilities of hypervisor memory operations.

A full list of features can be found on the Xen Community web site: http://xen.org.

lightning boltGNOME Project Updates Free Desktop with 2.30 Release

On March 31st, the GNOME Project celebrated the release of GNOME 2.30, the latest version of the popular, multi-platform free desktop environment and developer platform. Released on schedule, GNOME 2.30 builds on top of a long series of successful six month releases, including enhancements for user management, Web browsing, support for Facebook chat, and new productivity features.

GNOME 2.30 also marks the last major release in the GNOME 2.x era. There will be maintenance releases for 2.30, but the community now focuses its efforts on GNOME 3.0.

The 2.30 release contains significant user-visible improvements, adding numerous platform improvements for developers, and builds towards the upcoming GNOME 3.0 release with a preview of the revolutionary GNOME Shell. GNOME Shell, which will replace the existing GNOME Panel, changes the way users will interact with the desktop.

Also, this release includes a number of improvements in GNOME's Orca Screen Reader that improve performance and use on netbooks, and platform improvements to ready GNOME's Accessibility interface for GNOME 3.0. The 2.30 release also delivers comprehensive support for more than 50 languages, and partial support for many others.

"I'm really pleased with all of the updates in GNOME 2.30," said Stormy Peters, GNOME Executive Director. "I'm excited that I can automatically sync my Tomboy notes between my desktop and laptop computer, easily configure Facebook chat in Empathy instant messenger, and do more with PDFs in Evince. GNOME 2.30 provides everything I need for work and play."

GNOME 2.30 includes updates to Nautilus, the GNOME File Manager, including a new split view mode and is now set to browser mode by default, replacing spatial mode.

GNOME System Tools now works with PolicyKit for authentication, removing the Unlock button for managing users and services. The dialogue to create a user has been improved and only requires you to provide a the new user's name and will include user name suggestions. Upon account creation everything will "just work". This also includes support for encrypted home directories.

Vinagre, a remote desktop client for the GNOME Desktop, can now access to a remote machine more securely through an SSH tunnel. This needs an SSH account on the remote client.

Connecting to clients using a low bandwidth connection is eased by allowing you to choose a lower depth of colour to save bandwidth and by enabling JPEG compression. These options are available in the connection dialogue in Vinagre.

You'll find detailed information about GNOME 2.30 in the release notes: http://library.gnome.org/misc/release-notes/2.30/.

lightning boltOracle Announces MySQL Cluster 7.1

Demonstrating its investment in MySQL development, Oracle announced the latest release of MySQL Cluster at the MySQL User Conference in April.

MySQL Cluster 7.1 introduces MySQL Cluster Manager to automate the management of the MySQL Cluster database. In addition, the new MySQL Cluster Connector for Java enables higher throughput and lower latency for Java-based services.

MySQL Cluster is a leading "real-time" relational database with no single point of failure. By providing very high availability and predictable millisecond response times, MySQL Cluster allows telecom and embedded users to meet demanding real-time application requirements.

With support for in-memory and disk-based data, automatic data partitioning with load balancing, and the ability to add nodes to a running cluster with zero downtime, MySQL Cluster 7.1 delivers enhanced database scalability to handle unpredictable workloads.

MySQL Cluster 7.1 includes the following key features:

* MySQL Cluster Manager - allows users to manage a cluster of many nodes.
* NDBINFO - presents real-time status and usage statistics from the MySQL Cluster Data Nodes as SQL tables and views, providing developers and administrators a simple and consistent means of proactively monitoring and optimizing database performance and availability.
* MySQL Cluster Connector for Java - allows developers to write Java applications that can use JDBC or JPA to communicate directly with MySQL Cluster.

MySQL Cluster 7.1 can be downloaded today at http://dev.mysql.com/.


Talkback: Discuss this article with The Answer Gang


[BIO]

Deividson was born in União da Vitória, PR, Brazil, on 14/04/1984. He became interested in computing when he was still a kid, and started to code when he was 12 years old. He is a graduate in Information Systems and is finishing his specialization in Networks and Web Development. He codes in several languages, including C/C++/C#, PHP, Visual Basic, Object Pascal and others.

Deividson works in Porto União's Town Hall as a Computer Technician, and specializes in Web and Desktop system development, and Database/Network Maintenance.



Bio picture

Howard Dyckoff is a long term IT professional with primary experience at Fortune 100 and 200 firms. Before his IT career, he worked for Aviation Week and Space Technology magazine and before that used to edit SkyCom, a newsletter for astronomers and rocketeers. He hails from the Republic of Brooklyn [and Polytechnic Institute] and now, after several trips to Himalayan mountain tops, resides in the SF Bay Area with a large book collection and several pet rocks.

Howard maintains the Technology-Events blog at blogspot.com from which he contributes the Events listing for Linux Gazette. Visit the blog to preview some of the next month's NewsBytes Events.


Copyright © 2010, Deividson Luiz Okopnik and Howard Dyckoff. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Away Mission - Upcoming in May: Citrix Synergy and Google IO

By Howard Dyckoff

This year, Synergy plays in San Francisco, in the neighborhood of Silicon Valley and both VMware and Oracle (and also formerly independent Sun Microsystems.) Last year it was in Vegas, with keynotes available via Web streaming.

Synergy has become a new Virtualization destination since Citrix acquired the primary sponsorship of the Xen virtuallization platform from XenSource, Inc, in October 2007. It is also slowly developing Open Source tendrils.

It was only last year, a few months after Synergy 2009, that Simon Crosby, Citrix's CTO of Virtualization and Management, said, "XenServer is 100% free, and also shortly fully open sourced. There is no revenue from it at all." A further shift in the direction of Open Source is expected at May's event.

Synergy will also offer sessions on VDI - virtual desktop infrastructure. This seems follow along with Cloud Computing and the new internet-based terminal devices: smart phones and netbooks. We may be seeing the re-invention of the Mainframe.

If you’ve attended a previous Synergy conference, you’re entitled to a discount of $600 off the regular registration price. That would be only $1,295 and there is no deadline on that price. Other discounts are based on early registration.

At the Synergy Testing Center you can take a Citrix exam of your choice for free. This can help you pursue a Citrix Certification, including the CCEE and CCIA for Virtualization. The A15 and A16 beta exams are now available exclusively to Synergy San Francisco attendees.

Citrix Synergy-SF May 12-14, 2010. San Francisco, CA
http://www.citrix.com/lang/English/events.asp

I would also recommend Google I/O 2010; unfortunately, it is closed to new registrations. It filled up within 2 or 3 weeks of registration opening.

The 2010 web developer conference has expanded to include a BootCamp day before the main event, but you have to be already registered for Google IO to gain entrance to the BootCamp. These sessions assume you have no experience working with Google technologies and will be lead by Google developers.

Google I/O features 80 sessions, more than 3,000 developers, and over 100 demonstrations from developers showcasing their technologies.

The word must have gotten out that, besides a well organized conference and great grub, Google bestowed unlocked G-phones on all developer attendees last year. That's a big perk and some of these showed on eBay for over $500 each. Which meant several attendees got their fees covered because...

The cost last year and this year was fairly amazing. Early bird registration fees was $400 USD through April 16th, 2010. If there were space available after April 16th they would have been $500 USD. The Academia (students, professors, faculty/staff) registration fee was $100 USD. Such a bargain!

Of course, with the Iceland eruptions, some Euro guests may have to cancel. Send inquiries to googio2010@gmail.com and follow on twitter @googleio.

Google IO Developer Conference (now closed): May 19 - 20, Moscone Center, San Francisco, CA
http://code.google.com/events/io/2010/


Talkback: Discuss this article with The Answer Gang


Bio picture

Howard Dyckoff is a long term IT professional with primary experience at Fortune 100 and 200 firms. Before his IT career, he worked for Aviation Week and Space Technology magazine and before that used to edit SkyCom, a newsletter for astronomers and rocketeers. He hails from the Republic of Brooklyn [and Polytechnic Institute] and now, after several trips to Himalayan mountain tops, resides in the SF Bay Area with a large book collection and several pet rocks.

Howard maintains the Technology-Events blog at blogspot.com from which he contributes the Events listing for Linux Gazette. Visit the blog to preview some of the next month's NewsBytes Events.


Copyright © 2010, Howard Dyckoff. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Henry's Techno-Musings: My Help System

By Henry Grebler

Techno-Musings image

Preamble

If a doctor or an electrician came to your house empty-handed, you'd be a bit bemused, right? (I suppose, these days, if a doctor came to your house, you'd be astonished! But let's not get side-tracked.) You expect the professional and the tradesman to come with tools. How else is she going to do her job?

Yet, in the computer industry, people show up on their first day at work in a suit. And that's it. And, sometimes, not in a suit.

Strangely, not only are we not expected to bring our own tools, often the very idea of computer people relying on tools is considered a deficiency, a weakness. What?! You look phone numbers up in a directory! What a wuss! A real sysadmin would just memorise the phone book!

Context

I do not come to work empty-handed; I come with a large toolkit. Most of what I know fits into several classes. It also fits on a CD.

These are the components of my toolkit:

  1. HAL - Henry Abstraction Layer
  2. Muscle memory
  3. Help System
  4. History files

This article is about the third of these.

Motivation

Someone comes to me and asks, "What do you know about SAN, Henry?"

"I don't know," I reply. "Let's have a look."

	help san

I am presented with help on the Sandringham train timetable and nuisance calls, in addition to quite a lot of practical information on SAN accumulated over many years.

My help command is analogous to something which is a cross between apropos and some implementations of man. It has the wildcarding properties of apropos, but gives full answers like man rather than the headings returned by apropos.

It's different from man and complementary. man contains reference information; the information returned by help is more like what you might find in a HOWTO or a quick-start guide. man tends to contain complete information on a topic; the information returned by help is often fragmentary, incomplete and very basic.

Unlike man which is purely technical and only for the machine I'm on, help is not restricted to a single platform; nor is it restricted to technical matters. People who know me might be shocked to learn that I have help on Microsoft offerings. I also have help on how to use my bank accounts.

Invoking help

To use my help command, I type

	help string
where string is any string which I think might be relevant. For example,
	help rpm
should give me help on rpm.

My help directory

My help directory is $HOME/help. I was not very consistent at first. Arguably, I'm still not. From what I can see, my earliest entry is dated 14 Aug 1992.

These days, I work like this. Each entry is a file. Each file that I want to be able to look up has help_ as the first part of its filename. There may have been a reason for this, though, honestly, I don't recall. Perhaps I thought there could be other entries (non-text files like PDFs) that I would access some other way. I also had subdirectories. I guess I did not want them "getting in the way" when I was looking stuff up (so their names never began with help_). I don't use subdirectories very often these days; I can't recall the last time I did.

There's a subdir called FIREFOX which contains some html files; another called HTML which includes Quick_HTML_Reference.html; another called fedora-install-guide-en_US. But these subdirs are on the periphery of my help system.

The main game is the body of help_* files.

The "rules" are very simple:

My help command

My help command is implemented as a bash function defined in one of my many startup scripts (which form part of my HAL).
function help {
# Assume no arg means standard bash help

        if [ "$1" = "" ]
        then
                builtin help
                return
        fi
        echo 'For bash help, use builtin help'

# avoid backup files:   
        (set -x; less `ls ~/help/help_*${1}* | grep -v '~$'`)
        if [ $? -ne 0 ]
        then                    # try for bash help
                builtin help $1
        fi
}

The first few lines deal with the fact that I have appropriated bash's builtin help command. So, if I just type help, I get bash's help.

The line with the parentheses does the work. I'll come back to it.

The last 4 lines are supposed to help me coexist with bash. In theory, if I don't have help on the subject in my database, maybe I meant the bash command. In writing this up, I have discovered that this part doesn't work! This supports somebody's law that every non-trivial program has bugs.

It also supports the notion that something does not have to be perfect to be extremely useful. "Perfect is the enemy of good", as Ben said recently.

The main command

	less `ls ~/help/help_*${1}* | grep -v '~$'`

Let's start at the inside.

	ls ~/help/help_*${1}*

This lists all files in ~/help (ie $HOME/help) which start with help_ and have the specified string somewhere in the rest of the filename.

I use emacs as my editor. It creates backup files which have tilda (~) as the last character of the filename. I don't want to see these backup files; the grep removes them from the list.

The less command then displays the files left in the list, sometimes one file, often several.

Before I talk about my less command, I'll make one more comment about the line with the parentheses. It starts with set -x. Long ago, when I first met Unix (before Linux existed, I think), I read somewhere that Unix commands do their work quietly. That isn't how I like to operate; I like lots of confirmation: this is what I did, this is the command I used, these are the files on which I operated.

I don't expect others to share my preferences. Those who prefer silence can omit the set -x.

My 'less' command

Using less on several files would be vastly less useful without a little wrinkle.

I prefer to use less(1) as my $PAGER. However, standard less needs a little help. I want to be able to move from one file to the next and back quickly. By default, less uses :n and :p for this, but that's 3 keystrokes, and there is no way that 3 keystrokes can achieve what I want. I guess I could program the function keys to generate :n and :p, but there are issues with that.

I prefer to use the LEFTARROW and RIGHTARROW keys. By default, less uses these to mean "scroll horizontally", but I never want to be able to scroll horizontally in less, so I appropriate those keys.

I want RIGHTARROW to mean "next file" and LEFTARROW to mean "previous file".

If you want to do things my way, get hold of my .lesskey file and either copy it to your home directory or use it to edit your own .lesskey file - see lesskey(1). Either way, check what's there: you may not want everything I have. See below.

Then run lesskey to compile the .lesskey file into something less can use.

Analysis

In my view this is a wonderful example of appropriate technology. It's appropriate when all factors are taken into consideration. The most important factor is the customer. I designed this to be used by me, no one else. Of course, "design" is a gross overstatement; the mechanism grew organically.

Most of my interaction with computers is via the command line, typically running in an xterm. Whenever I can, I use bash. That's pretty nearly always: bash is the default shell on Linux (that which we think of as /bin/sh is really bash); it is now standard (/bin/bash) on recent versions of Solaris; and is readily available on BSD. When I'm not typing commands, it's probably because I've invoked emacs.

Whether I'm in emacs or in bash, most of the commands I use are the same.

So, editing a help file involves one of my bread-and-butter activities; nothing new to learn.

Probably the next most common thing I do is use less to browse files. It is my mail reader of choice (unless I absolutely can't use less). Consequently, reading my help files also involves one of my bread-and-butter activities.

Most importantly, it's wonderfully portable. I can take the whole help directory (and other stuff) on a CD wherever I go and copy it to local disk in no time.

The system is deliberately as unstructured as possible. There is a danger that if I make entering information too onerous, I won't do it. In many organisations, generally helpful information must be entered in some markup language. If I have to organise my help information at the time I'm recording it, I'm going to have to suspend doing whatever task I'm working on. That won't do.

Typically, I just swipe and paste text from where I'm working into a file. I'm just creating aids for myself - the equivalent of rough notes one might write on a scrap of paper.

If, some time later, I return to a help file because I'm working in the same area, I might clean the information a little. Or not.

The major purpose is to save time. If I spend a long time trying to work out how to do something, I don't want to have to repeat that exercise in the future.

Of course I'm human and fallible. Sometimes I'm so happy that I got the thing to work, I just jump in and use it; I forget to record useful information. But, hopefully, I'll catch it next time.

Typical use

Let's take an example. I've recently moved to FreeBSD. I'm a bit of a newbie. Let's see what I feel I need to know.

	ls -lat $HOME/help/help_*bsd* | grep -v '~' | cut -c 30-
 3897 Mar 21 07:35 /home/henryg/help/help_freew_bsd_diary
 3030 Mar 19 14:50 /home/henryg/help/help_bsd_build_notes
  100 Mar 19 11:12 /home/henryg/help/help_bsd
  113 Mar 17 16:38 /home/henryg/help/help_bsd_admin
  261 Mar 17 10:37 /home/henryg/help/help_bsd_build

So when I go help bsd, the effect is to load the above 5 files into less. Because I can use the left- and right-arrow keys to navigate from one file to the next, and because these keys auto-repeat, I can get to any file very quickly. Since I put this stuff in, it follows that I know what's there (more or less). The way my brain works these days, I'm much better at recognising and selecting than I am at recalling.

If I'm looking at a file with less, and I decide I want to modify the file in some way, I can simply type v and I'm in emacs ready to edit.

If all else fails, I can grep through the help files since they are just text files. Or I can list the help directory to see if any of the filenames trigger recognition.

This rarely happens, but let's say I'm looking for some info on xxx and help xxx doesn't produce anything useful. I resort to grep and find the info. Why didn't help xxx deliver the relevant file? I realise that I have "misfiled". For instance, I stored some info in help_audio but I subsequently look for it with help sound. Ok, simple solution:

	mv help_audio help_audio_sound
I can now find the relevant help by using either string (or a substring). For instance, either of these commands will deliver the above file (and possibly others):
        help aud
        help soun

I can "file" some information under a dozen categories (strings) virtually for free. I simply need to include more items in the name of the help file.

The contents of help files

Please remember that the information in a help file is what I believe will be useful to me at some time in the future. If you are interested in such a mechanism for yourself, I expect you to put into your help files the sort of information that you will find useful. That could be quite different.

Just so you get the flavour, here's the first part of help_rpm:

### For complicated stuff see help_rpm2
### For Uninstall see help_rpm_uninstall
------------------------------------------------------------------------

RPM=wxpythongtk-2.4.2.4-1.rhfc2.nr.i386.rpm

rpm --query -p $RPM   -i
rpm --query -p $RPM   -l | cut -d / -f1-3 | uniq | head
rpm --query -p $RPM   -l | wc
rpm --query -p $RPM   -l

# Test Install
#                               rpm -i --test $RPM      # Use the long option
rpm --install --test $RPM
# Really install
#                               rpm -i $RPM             # Use the long option
rpm --install $RPM

++++++++++++++++++++++++++++++++++++++++++++++++
Tough installs for extreme situations:

rpm -i $RPM --force
rpm -i $RPM --nodeps            # Don't check dependencies

Typically, to perform an install, I set the variable RPM, then copy and paste each of the rpm commands into an xterm. If I want, I can pretty much turn my brain off and run on autopilot.

More importantly, it saves me a lot of typing; it minimises typos; and it allows me to concentrate on more global matters, not the mechanics of an install.

One more example: here's help on the string pub. (Yours might tell you about watering holes.) I have two help files concerned with public transport: help_public_transport and help_aaa_public_transport. I created the first one first. It has some detailed information on bus routes. It's the one I usually don't want to see. So, when I created the second file I put aaa at the start of the string so that it will always show as the first file in less (because of the way ls works).

When I want to go somewhere by public transport, I visit Viclink Journey Planner. It's a marvellous resource, but it can be a little too general at times. To get answers more quickly, it helps to provide detailed strings, strings that I would never bother to remember and could probably never type correctly. So my help shows me the strings I need for the stations I frequent:

Gardenvale Railway Station (Brighton)

Ormond Railway Station (Ormond)
Caulfield Railway Station (Caulfield East)
Malvern Railway Station (Malvern) 
Armadale Railway Station (Armadale) 

Southern Cross Railway Station (Melbourne City) 
Richmond Railway Station (Richmond)
Flinders Street Railway Station (Melbourne City)
Melbourne Central Railway Station (Melbourne City)

I just paste the relevant strings into the form and get answers very quickly.

Extensions

After many years working with one organisation, recently I've been contracting to different companies. I am just in the process of working out how I want to handle this. I'm thinking that the stuff I gather while working at X Corp could go into directory help_X_Corp and the Y Corp stuff could go into help_Y_Corp. I started doing something along these lines at my last contract.

I'm leaning towards formalising this, using a modified form of the help command. The key line is:

	               v
	less `ls ~/help*/help_*${1}* | grep -v '~$'`
	               ^

I've added an extra asterisk to wildcard across ALL help directories.

In Conclusion

That's it in a nutshell. Anyone can adapt what I've done to suit him/herself. That's the beauty of computers and especially open source software: near infinite flexibility and customisation. It's such a shame that we waste the power on "look and feel" (skins and themes).

"Off the rack" fits most people reasonably well without fitting anyone perfectly. But anyone can adapt a system like this to suit as closely as desired - with enough effort.

It doesn't matter if you are a computer nerd or a neophyte. It doesn't really matter what OS you use (though Microsoft makes it harder). You need an editor and some way to select and display files. The only difficulty I can see is if your only access is through GUIs, because then you have to write (or get someone to write) the selection code (and maybe the display code). I guess the Microsoft way is to also use the editor (say Wordpad) as the display program. I'm not real keen on that, but others might prefer it.

If I'd wanted that functionality, I could have used emacs in place of less. I think I would have lost a lot if I'd done that.

Postscript

I wrote the above a few days ago. The next day, pricked by some of the points I made, I modified my help command. I've included it in Resources below.

Resources

#	.lesskey - less keybinding file

#  Henry Grebler    26 May 98  Add alternative left-scroll.
#  Henry Grebler    26 May 97  First cut.
#=============================================================================#

# Note on comments:

#     Blank lines and lines which start with a pound sign (#)  are
#     ignored, except for the special section header lines.

#=============================================================================#
# Note on usage:

# This file is the input file to lesskey. Without args, lesskey
# processes $HOME/.lesskey (a text file) to produce $HOME/.less (a
# binary file (which is the init file for less)).

#=============================================================================#

          x         quit
          X         quit
          \e[C        next-file
          \e[D        prev-file
          \eOC        next-file
          \eOD        prev-file

# By default, \e[ is left-scroll. However, the inclusion of \e[C above
# seems to have disabled \e[ (sort of understandable). So I add \e{ as
# a synonym for left-scroll (and add \e} as a synonym for right-scroll
# as an afterthought for consistency).

#          \e{         left-scroll
#          \e}         right-scroll

My latest help command:

function help {
# Assume no arg means standard help

        if [ "$1" = "" ]
        then
                builtin help
                return
        fi
        echo 'For bash help, use builtin help'

# avoid backup files:
        files=`ls ~/help*/help_*${1}* 2>/dev/null | grep -v "~$"`
        if [ "$files" = '' ]
        then                    # no matches: try for bash help
                builtin help $1
                return
        fi
        echo "$files"
        echo 'ls ~/help*/help_*'${1}'* | grep -v "~$"'
        less $files
}


Talkback: Discuss this article with The Answer Gang


[BIO]

Henry has spent his days working with computers, mostly for computer manufacturers or software developers. His early computer experience includes relics such as punch cards, paper tape and mag tape. It is his darkest secret that he has been paid to do the sorts of things he would have paid money to be allowed to do. Just don't tell any of his employers.

He has used Linux as his personal home desktop since the family got its first PC in 1996. Back then, when the family shared the one PC, it was a dual-boot Windows/Slackware setup. Now that each member has his/her own computer, Henry somehow survives in a purely Linux world.

He lives in a suburb of Melbourne, Australia.


Copyright © 2010, Henry Grebler. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Pixie Chronicles: Part 2 Bootable Linux

By Henry Grebler

The Story So Far

In Part 1, I said that my project was to build a server. However I glossed over many details. In this part, I propose to examine those details more closely. I also raised a problem, which I left hanging. I'll get back to it a bit later.

By the end of this set of articles, you should have enough information to perform your own PXE network install.

Now Read On ...

Gathering Information

Before I install anything, I like to get some idea of what is on the machine. Does it have enough memory? How big is the hard drive? Will Linux recognise all the hardware? For this task, nothing could be better than a Linux which can run off a CD. I have a couple of versions of Knoppix, a couple of versions of Ubuntu and a Fedora which fit this bill quite nicely. I'm sure there are others.

My preference is for Knoppix. It has one nice capability that does not seem to be available on Ubuntu: once the machine has booted from the CD, it is a simple task to configure networking and then start sshd (the SSH daemon). I can then return to my desktop, ssh into the target machine and do everything from the comfort of "home". "Home" has my favourite keyboard, mouse and screen. When the target machine is rolled out, I will probably choose to run it headless, keyboardless and mouseless. However while it is being built, I connect the smallest, oldest screen, keyboard and mouse I have lying around They work, but they are not comfortable.

I had been given a machine with a single NIC. I inserted a second, a spare which I'd bought at a swap meet and which was lying around waiting to be useful.

For the network install, I would need the MAC address (aka Ethernet address) of at least one NIC, preferably both. This is exactly the sort of information that one can easily get if one first boots a standalone system from CD.

There are other uses for bootable Linux (or Live Linux CD). Some friends and I have developed a very powerful mechanism for cleaning viruses from Windows machines. When a machine is infected so badly that the viruses take control before Windows, a bootable Linux CD and a USB drive can save the day.

Investigation

And now for the use which is the point of this part. Part 1 ended with the discovery that I had installed Fedora 10, the target machine had then rebooted and restarted the install. It had stopped waiting for clarification about partitioning.

Ok, so what state was my target machine in?

I booted off the Knoppix CD to look around. What was on the hard drive? Hmm, nothing. That's a bit disappointing. What about all the software that I had seen it install? Actually, it was less nothing than that. The partitioning information had disappeared!

I remembered something that had been in the kickstart file. Or thought I remembered. I decided to check.

Here is a huge advantage of using kickstart as opposed to interactive install. If there is doubt about what responses were given at install time, with an interactive install one can only rely on one's memory. And in some cases, only on someone else's memory. I have oodles of proof that my memory is fallible; and also that many people have merely human memory. Even if one is assiduous, as I often am, and one writes down one's responses, that only makes the situation better; it's still not perfect. But the kickstart file can always be examined to confirm or disprove a suspicion.

I looked at the kickstart file. There was an interesting line:

	clearpart --all --initlabel

Clearly, I hadn't been paying enough attention when I was constructing the kickstart file. (I'll have more to say about this at another time.) The line requests that all disks have their partition tables removed and their labels reinitialised.

All?!

[Uneasy feeling.] Surely not ALL?! Surely only real hard drives.

It's Worse Than I Thought

To make my life more comfortable, when I boot off the Knoppix CD, I usually insert my USB stick as well. Apart from the antivirus stuff, the USB stick has my environment - my aliases, functions and scripts. These are my equivalent of a tradesman's tools of trade. With a couple of commands, I can recreate my personal environment. It also helps to shield me from the variations between different Unix and Linux systems: I have created a sort of HAL - Henry Abstraction Layer.

I examined my USB stick. All gone! The install had cleared its partition table. On reflection, that probably occurred on the first cycle through the install process, and was repeated on the second. A subtle but hardly comforting distinction.

In addition to losing the install, I had lost all the information on my USB stick!

If you've been paying attention, you might have guessed that all is not lost. In Part 1 I said, "... it's important to be able to recover from mistakes."

Fortunately I am a wizard of not inconsiderable power and I was able to recreate the partition table on the USB stick.

Analysis

I was also able to restore the install on the hard drive. But that's not where I want to go right now. I want to pick up on the last point of Part 1.

I implied that just changing the kickstart file to not reboot at the end of the install was short-sighted. Here's why.

Making that change certainly stops problems at the end of the install. But what happens if I turn off the machine for whatever reason and then turn it back on again? You might say that if I behaved like that I would be foolish and would deserve what I got. True. But, what if I turned off the machine and someone else turned it on? I need a robust solution.

Ideally, there ought to be a way to perform the PXE boot just once. I'm not aware of such a mechanism on my hardware. When one configures a boot sequence in a BIOS, one usually sets CD before hard drive. If a bootable CD is in the drive, the machine boots from CD. If not, it boots from hard drive. And it's true that if there is no network, the machine will give up on a PXE boot and try the CD and hard drive. But that doesn't help me. I need the network there for the network install. I would then have to remember to remove the network before the next reboot. I have not solved the problem.

For my purposes, it would be nice if the PXE boot mechanism required that the user press a key while the machine was booting. So, when I first turned on the machine, I would hold the key down and the PXE boot would take over. Once the PXE sequence had started, I would stop pressing the key. Now if the machine reboots at the end of the install, or it is switched on in the usual way, the PXE boot is bypassed.

Dream on. I needed to look elsewhere. I talked to a colleague, Ian, who pointed me in the right direction.

During the PXE boot, the machine downloads the installation environment and a PXELINUX config file (which is very similar to a GRUB config file). The PXELINUX config file can be set to simply proceed; alternatively it can be set to offer the user a number of choices for how to boot.

After some judicious editing, my PXELINUX config file now achieves a very serviceable compromise. When invoked it presents a boot: prompt. If the user types b2 (the name of this machine) within 10 seconds, the kickstart install process is invoked. There are some other nice options but I'll defer discussing those.

If the user presses Enter or 10 seconds elapse with no user input, the machine boots from local hard disk. Now it doesn't matter how the machine is turned on: the default condition is to boot from local hard drive. I have to be at the keyboard to start the kickstart process, but I'm there anyway to turn on the machine. I have to wait for perhaps a minute for the PXE boot to finish and the boot: prompt to appear. I then enter b2 to start the kickstart install - and then I can leave. The rest of the install will be automatic. If I've requested a reboot at the end of the install, the machine will perform the PXE boot from the network and after 10 seconds boot off the hard drive. All boxes ticked.

Eventually, when I'm satisfied that the machine is set up how I want it, I can get into the BIOS and turn off the PXE boot.

Lessons

Amongst its many advantages, a kickstart file provides an audit trail and reproducibility. It also allows for convenient change management.

One can always keep track of changes using documentation, but it requires more discipline and is more error-prone.

Look for robust solutions.

Consider many scenarios, not just the one which presented the problem.

Reference

For more on Knoppix see this article: Knoppix gives bootable, one-disk Linux


Talkback: Discuss this article with The Answer Gang


[BIO]

Henry has spent his days working with computers, mostly for computer manufacturers or software developers. His early computer experience includes relics such as punch cards, paper tape and mag tape. It is his darkest secret that he has been paid to do the sorts of things he would have paid money to be allowed to do. Just don't tell any of his employers.

He has used Linux as his personal home desktop since the family got its first PC in 1996. Back then, when the family shared the one PC, it was a dual-boot Windows/Slackware setup. Now that each member has his/her own computer, Henry somehow survives in a purely Linux world.

He lives in a suburb of Melbourne, Australia.


Copyright © 2010, Henry Grebler. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Solving sudokus with GNU Octave (I)

By Víctor Luaña and Alberto Otero-de-la-Roza

I. Introduction

GNU Octave is a high level language designed for science and engineering applications. Easy to learn and simple to teach, Octave can be used as a sophisticated interactive graphical calculator. On the other hand, the rich set of data structures (complex numbers, vectors, matrices, multidimensional arrays, strings, lists, function handles, cell arrays, ...) and, in particular, the complete collection of matrix operations and functions give Octave enough punch to be used in professional applications. From a programmer's point of view, the interface of Octave functions is superb. Incidentally, Octave is highly compatible in notation and functionality with MatLab, a leading instrument in many engineer's toolbox.

The intention of this article is to show the matrix capabilities of the language by analyzing the solution of arbitrary 9x9 classical sudokus. A primitive code, combining simple scan and brute force trial-and-error guessing will be discussed. More advanced solving techniques will be the subject of a future installment.

II. Matrix operations

Before going on to the sudoku problem, it may be useful to discuss a few ideas about the language matrix capabilities. First of all, Octave matrices can be defined and addressed element by element, but it is far more efficient to handle them as a whole using the rich set of operations and functions that the language provides. Compare, for instance, the following code for defining and multiplying two 100x100 random matrices element by element:

tstart = cputime;   # Start timing
rand("seed",pi);    # Initialize random number to a known seed
for i = 1:100       # Fill up the two 100x100 random matrices
   for j = 1:100
      A(i,j) = rand();
   endfor
endfor
for i = 1:100
   for j = 1:100
      B(i,j) = rand();
   endfor
endfor
for i = 1:100       # Multiply the matrices
   for j = 1:100
      C(i,j) = 0;
      for k = 1:100
         C(i,j) += A(i,k) * B(k,j);
      endfor
   endfor
endfor
Sum = 0;            # The sum of all elements in C
for i = 1:100
   for j = 1:100
      Sum += C(i,j);
   endfor
endfor
disp(Sum);
tend = cputime() - tstart;
disp(tend);

with the equivalent code using matrices at once

tstart = cputime;   # Start timing
rand("seed",pi);    # Initialize random number to a known seed
A = rand(100);      # Fill up the two 100x100 random matrices
B = rand(100);
C = A * B;          # Multiply the matrices
Sum = sum(sum(C));  # The sum of all elements in C
                    # The inner sum() returns the sum of each column
disp(Sum);
tend = cputime() - tstart;
disp(tend);

The matrix code is shorter, simpler, and some 2500 times faster (0.024 seconds compared to 58.8 seconds, on a typical 2.4 GHz Pentium). The main reason for this huge CPU time difference lies in the fact Octave is an interpreted language, so while loops must be parsed, interpreted, and performed once per cycle, matrix operations are transformed at once into fast machine code able to exploit the vector capabilities of the CPU.

Addressing parts of a matrix is simple with Octave. The i-th row and the j-th column of M are invoked as M(i,:) and M(:,j), respectively; M(i:i+3,j:j+3) slices a 4x4 block that starts at the M(i,j) element; M(2:2:,1:2:) slices all elements that belong to an even row and an odd column; and M(find(M>0)) returns a vector made of all elements in M that are positive. More matrix magic can be done by using address indices.

As a final remark on the specifics of Octave matrices, the elements are stored in column-major order (i.e. Fortran-like order, with M(1,1) being followed by M(2,1), M(3,1), etc.), and the starting value for the indices is 1 by default. As a default, a two-dimensional matrix can also be addressed as a one dimensional vector but if that appears a little confussing, the reshape() function can be used to explicitly change the matrix dimensions.

III. The first sudoku solver

A classical sudoku is a 9x9 matrix formed by nine 3x3 blocks. The 0-9 digits fill the 81 matrix cells in such a way that each digit appears exactly once in each row, each column, and each block. In fact, the name sudoku comes from the Japanese phrase "suuji wa dokushin ni kagiru" ("the numbers must occur only once") that was used by the Nicoli publishing house. Starting from a blank matrix, there are 6,670,903,752,021,072,936,960 different possibilities (Felgenhauer and Jarvis, 2005), that reduce to 5,472,730,538 unique sequences when all puzzle symmetries are taken into account (Russell and Jarvis, 2005). The mathematical enumeration becomes an interesting logical mind game when a few clues or given digits are added at selected places in the matrix. In the sudoku jargon, a proper puzzle has a unique solution, whereas a satisfactory puzzle is proper and can be solved without trial and error.

Two different types of methods have been applied in solving sudokus by computer. The first kind of programs try to follow the same or similar logical steps that human solvers employ. The second approach is based on a reformulation of the problem as the minimization of a penalty function. The logical way has two appealing advantages for our purposes: it can trace the reasoning used in a puzzle, and the logical steps can be used to grade the difficulty from a human point of view.

The code to be introduced in this first article is very basic on purpose. It uses a minimum of logical reasoning, relying heavily on trial and error guessing. The input/output is also minimalistic. Notwithstanding its limitations, the code (sudoku01.m from now on) is able to find all possible solutions of a given puzzle entered by the user. Let me show first the code and go later to discuss its main parts.

#! /usr/bin/octave -q
#
# Usage: sudoku01.m [datafile]
# Solves the sudoku puzzle contained in datafile (sudoku.dat as a default).
# The datafile contains the puzzle in any of the following formats:
#  * a 9x9 matrix of integers, with zeros on the unknown positions,
#  * a string of 81 chars, with zeros or dots on the unknown positions.
#

function b = blocknum(r, c)
   % Returns the block number for row r and column c
   b = 3 * floor((r-1)/3) + floor((c-1)/3) + 1;
endfunction

function sud = str2digits(str)
   % Converts a 81 digit string into a 9x9 integer matrix
   sud = zeros(length(str),1);
   for i = 1:length(str)
      sud(i) = str2num(str(i:i));
   endfor
   sud = reshape(sud,9,9);
endfunction

function sudoku_print (remark, sud)
   % Print a sudoku puzzle with some label text before
   disp(remark); disp(sud);
endfunction

function sud = sudoku_read(narg, arglist)
   % Reads in sudoku from input file
   if (narg<1)
      datafile = "sudoku.dat";
   elseif (narg==1)
      datafile = sprintf("%s", arglist(1));
   else
      error ("Use: octave -q %s [datafile]\n", program_name());
   endif
   lud = fopen(datafile, 'r');
   line = fgetl(lud);
   fclose(lud);
   if (length(line) >= 81)
      strrep(line,'.','0');
      sud = str2digits(line);
   else
      sud = load(datafile);
   endif
   sudoku_print('Starting puzzle:', sud);
endfunction

function [solved,sudout] = sudoku_solve(sud)
   global nsol
   % sudoku_solve - Starts the sudoku analysis, detects trivial cases,
   % and calls in the real solvers.
   solved = 0; sudout = sud;
   row = zeros(9); column = zeros(9); block = zeros(9);
   for r = 1:9
      for c = 1:9
         if (sud(r,c))
            n = sud(r,c); b = blocknum(r,c);
            if (row(r,n) | column(c,n) | block(b,n))
               solved = 0; return
            endif
            row(r,n) = 1; column(c,n) = 1; block(b,n) = 1;
         endif
      endfor
   endfor
   unknown = length( find( sud==0 ) );
   if (unknown > 0)
      [solved,sudout] = sudoku_trial(sud,row,column,block);
   endif
endfunction

function [solved,sudout] = sudoku_trial(sud,row,column,block)
   global nsol ncall cpu0 levcall levmax
   % sudoku_trial - Minimal recursive solver
   ncall++; if (++levcall > levmax); levmax = levcall; endif;
   solved = 0; sudout = sud;
   % First step: scan for forced positions
   do
      new = 0; minpos = 9;
      for r = 1:9
         for c = 1:9
            if (sud(r,c)==0)
               b = blocknum(r,c);
               pos = row(r,:) | column(c,:) | block(b,:);
               ipos = find(pos==0);
               npos = columns(ipos);
               if (npos==0)
                  solved = 0; levcall--; return
               elseif (npos==1)
                  n = ipos(1); sud(r,c) = n; new++;
                  row(r,n) = 1; column(c,n) = 1; block(b,n) = 1;
               elseif (npos < minpos)
                  minpos = npos; minipos = ipos;
                  minr = r; minc = c; minb = b;
               endif
            endif
         endfor
      endfor
   until (!new)
   unknown = length( find( sud==0 ) );
   if (unknown==0)
      sudoku_print(['Solution num. ' num2str(++nsol)
           'cpu ' num2str(cputime-cpu0) ', call ' num2str(ncall) \
           ', rec ' num2str(levcall) ', max ' num2str(levmax)], sud);
      solved = 1; sudout = sud; levcall--; return
   else
      % Second step: guess a digit for the best position found
      for p = 1:minpos
         n = minipos(p); sud(minr,minc) = n;
         row(minr,n) = 1; column(minc,n) = 1; block(minb,n) = 1;
         [res2, sud2] = sudoku_trial(sud,row,column,block);
         row(minr,n) = 0; column(minc,n) = 0; block(minb,n) = 0;
         if (res2)
            solved = 1;
            sudout = sud2;
         endif
      endfor
      levcall--;
   endif
endfunction

%%%%%%%%%%%%%%%%%%%%%%% Main code starts here %%%%%%%%%%%%%%%%%%%%%%%
% Use global variables to get some statistics on the solving method.
global nsol=0 ncall=0 cpu0 levcall=0 levmax=0
sud = sudoku_read(nargin, argv);
cpu0 = cputime();
[solved, sud] = sudoku_solve(sud);
if (solved==0)
   sudoku_print("Unable to solve the puzzle!", sud);
endif
printf ("total cpu: %g\n", cputime()-cpu0);
printf ("Trial calls and recursion depth: %d %d\n", ncall, levmax);

Given as input the well-known top1465/number77 case:

700000400020070080003008009000500300060020090001007006000300900030040060009001035

sudoku01.m took 2.2 sec. on a busy 1.8 GHz Pentium 4 CPU to find the solution, although it expended 17 more sec. fully exploring the solution tree until finally finding that it was unique. Here is the complete output:

Starting puzzle:
   7   0   0   0   0   0   0   0   0
   0   2   0   0   6   0   0   3   0
   0   0   3   0   0   1   0   0   9
   0   0   0   5   0   0   3   0   0
   0   7   0   0   2   0   0   4   0
   0   0   8   0   0   7   0   0   1
   4   0   0   3   0   0   9   0   0
   0   8   0   0   9   0   0   6   3
   0   0   9   0   0   6   0   0   5
Solution num. 1
cpu 2.164, call 46, rec 8, max 12
   7   1   4   9   5   3   6   8   2
   9   2   5   7   6   8   1   3   4
   8   6   3   2   4   1   7   5   9
   6   9   2   5   1   4   3   7   8
   3   7   1   8   2   9   5   4   6
   5   4   8   6   3   7   2   9   1
   4   5   6   3   8   2   9   1   7
   2   8   7   1   9   5   4   6   3
   1   3   9   4   7   6   8   2   5
total cpu: 19.2492
Trial calls and recursion depth: 435 15

Comparing these statistics with future versions of the code will make an interesting test of the usefulness of human-like logic versus brute-force exploring.

A. The input/output part

The input/output part of sudoku01.m is extremely simple. The code expects a data file name from the command line:

octave -qf sudoku01.m top1465n0077.dat

but the default, sudoku.dat, is used if no arguments are given. The data file contains the input puzzle in any of three formats:

The output is managed by the sudoku_print() local function, minimally reduced to a pair of disp() calls. The input is handled by sudoku_read(), which uses argv() to retrieve the program arguments, fgetl() to read a string, or load to read the matrix format, and the local function str2digits() to convert a string, character by character, into a matrix of digits.

B. Keeping the principal information ready

The current state of the puzzle is a 9x9 matrix of integer digits, with zeros filling in the cells yet unknown. This matrix is appropriate for simple tasks like obtaining the number of unknown cells, length(find(sudoku_matrix==0)), or printing, disp(sudoku_matrix). Rows, columns, and blocks in the sudoku matrix are numbered according to figure 1.

A test sudoku, labelling rows, columns and blocks
Figure 1 A standard 9x9 sudoku showing how rows, columns, and blocks are labelled.

The analysis of the puzzle is facilitated through the use of three 9x9 logical matrices: row(,), column(,), and block(,). The three of them have a similar structure and meaning. For instance, block(b,) is a column vector of nine elements that contain a one (meaning 'true') in the positions that correspond to the digits already assigned in block b, and a zero (meaning 'false') in the position of the digits that are not yet assigned. So, in the case of the sudoku shown in figure 1, block(3,:) is [0 0 1 1 1 0 0 0 0] because 3, 4, and 5 are the only digits already assigned in block number 3.

The row(,), column(,), and block(,) are initially created in the sudoku_solve() function, but they are used and maintained in synchrony with the sudoku matrix in sudoku_trial(), the real core function of the program and the true puzzle solver.

C. The sudoku_trial() function

This principal function of the solver is made up of two different parts. The first part determines the number of digits admissible for every unknown cell, identifies and fills those cells that have a single candidate, and repeats this process until no new information is gained in a complete cycle. The second part then takes an educated guess about the content of a cell and recursively calls the sudoku_trial() function to check whether the guess produces a correct solution for the sudoku or gives rise to an impossible situation. Let me describe both steps in a little more detail.

D. Naked singles scanning

Determining the possible candidates for a cell is simple with the help of the row(,), column(,), and block(,) matrices. In the case of the cell in row r and column c that belongs in block b, the logical expression

   row(r,:) | column(c,:) | block(b,:)

where | is the logical OR operator, produces a vector with zeros in the position of the digits that still can occupy the cell. The following table shows the result for the cell (r = 3, c = 7) of the test sudoku in figure 1:

[ 1 2 3 4 5 6 7 8 9 ]
row(3,:) [ 0 0 1 0 1 0 1 1 0 ]
column(7,:) [ 1 0 0 0 0 1 0 0 0 ]
block(3,:) [ 0 0 1 1 1 0 0 0 0 ]
OR result [ 1 0 1 1 1 1 1 1 0 ]

Two and nine are the two digits still admissible for the cell of the above example. In general, the outcome of this analysis for any cell gives rise to three different cases: (a) when there is a single candidate for a cell we can assign it immediately, taking care of actualizing the state of the sudoku matrix but also of the row(), column(), and block() matrices; (b) when there is no admissible candidate, we have arrived at an impossible state for the puzzle, probably as a result of an incorrect guess in a trial-and-error search, and the solving routine must be terminated with an error condition; and (c) when there are two or more possible candidates. At that point, nothing can be done, but the cell with the least number of multiple candidates is remembered for the second part of the routine.

The assignment of a value to a cell produces information that directly influences all cells that belong in the same row, column, or block, and indirectly the neighbors of these neighbors. Therefore, the search of single candidate cells (naked singles or singletons in the parlance of some sudoku aficionados) must be repeated until no new assignment is done in a whole cycle. This loop structure will remain similar when more advanced scanning techniques are introduced in our code.

E. Trial and error guessing

Eventually, the scan loop arrives at a point where no further information is gained. This is where guessing takes control. The cell with the smallest number of candidates has already been identified in the previous scan phase: minr, minc and minb are the row, column, and block of this pivotal element, minpos is the number of admissible digits, and minipos() is a vector containing the actual digits.

The trial and error step consists of a loop that tries in sequence all minipos(1:minpos) admissible digits for the pivot. The state of the sudoku before this loop must be conserved so that each candidate digit starts fresh from the same situation. The puzzle state is then changed for the new candidate, and control is passed recursively to the sudoku_trial() function for a new turn of scanning plus guessing. The recursion stops when either a solution is found or an impossible puzzle state is reached. An infinite recursive loop is not possible because every new call to sudoku_trial() contains at least one unknown cell fewer than the previous level call.

Keeping the puzzle state between alternative candidates is, in fact, a consequence of the way that parameters are passed to Octave functions. Octave function arguments are passed by value, which means that each argument is evaluated and copied to a variable local only to the function. Unlike most languages, however, Octave functions can return multiple values to the calling program block. This behavior is exploited in the following fragment of our recursive routine:

function [solved,sudout] = sudoku_trial(sud,row,column,block)
   ...
   for p = 1:minpos
      ...
      row(minr,n) = 1; column(minc,n) = 1; block(minb,n) = 1;
      [res2, sud2] = sudoku_trial(sud,row,column,block);
      row(minr,n) = 0; column(minc,n) = 0; block(minb,n) = 0;
      if (res2) solved=1; sudout=sud2; endif
      ...
   endfor
   ...
endfunction

The sud(), row(), column(), and block() matrices can be changed within the function, with confidence that their modified value will not be passed back.

IV. Coming next

The naked singles scanning is just the simplest of the several logical techniques that have been proposed to solve the sudoku puzzles. In future articles, we will examine how to implement several additional methods: hidden singles, subset interactions, x- and xy-wings, swordfish, etc. The final objective is trying to solve as many puzzles as possible using direct or indirect logical reasoning and avoiding the trial and error step.

V. MatLab versus octave

To run the code discussed here under MatLab, please change all occurrences of the specific terminators endfor, endif, and endfunction into the generic end. Octave, but not MatLab, recognizes both types of endings. Hopefully no further changes should be needed, but we would not be surprised to learn otherwise.

Acknowledgements

We acknowledge support by the Spanish Ministerio de Ciencia e Innovación (MICINN) and the ERDF of the European Union (project no. CTQ2009-08376). The authors belong to the MALTA group (CSD2007-0045 project, MEC Consolider Ingenio 2010 program). AOR is indebted to the Spanish MEC for an FPU grant.

References

  1. Bertram Felgenhauer and Frazer Jarvis, Enumerating possible Sudoku grids, http://www.afjarvis.staff.shef.ac.uk/sudoku/sudoku.pdf, June 2005 (Last retrieved on feb. 2010). See Jarvis sudoku page for more related results.
  2. Ed Russell and Frazer Jarvis, Mathematics of Sudoku II, http://www.afjarvis.staff.shef.ac.uk/sudoku/russell_jarvis_spec2.pdf, January 2006 (Last retrieved on feb. 2010).
  3. English wikipedia, List of Sudoku terms and jargon, http://en.wikipedia.org/wiki/List_of_Sudoku_terms_and_jargon. Last retrieved on feb. 2010.
  4. Angus Johnson, Simple sudoku, http://www.angusj.com/sudoku/hints.php (Last retrieved on feb. 2010).
  5. Top1465 is a compilation of difficult sudokus produced by Guenter Stertenbrink (aka dukuso), based on the rates given by the suexratt (http://magictour.free.fr/suexratt.exe) solver. The collection itself can be downloaded from http://magictour.free.fr/top1465 (Last retrieved Mar 2010).

Talkback: Discuss this article with The Answer Gang


[BIO]

I'm a Spaniard, living in Oviedo, close to the northern coast of Spain. Married, with a teenager daughter ("hey, dad, this spiro thing is cool!"), I teach Physical Chemistry and do research on Theoretical Chemistry for a living. I have programmed on all sort of machines, from the mythical HP41C to the not less mythical Cray X and Y supercomputers (the end of an era left us short of a good crazy machine), for the last 25 years. I was captivated by linux around 1993, when our research group discovered the first Slackware distribution. Our current infrastructure is based on Debian and includes a half-mounted Top500 wannabee shared among many research groups.

On the left: self portrait in front of a mirror (Feb. 2006). Yes, I truly love photography, but I don't like appearing on images myself. Anyway, this self portrait has something that I don't dislike too much. The band on my left hand has disappeared long ago.



[BIO]

I'm currently working in Oviedo, completing my Ph.D. in chemistry, near my birth place, a small town nearby called La Felguera. My interests include a wide range of fields but I am specially keen on (hardcore) scientific programming and scripting languages, with application to chemical problems in solids. When I'm not working, I enjoy reading and gaming (anything that tells a good story) and also playing chess.


Copyright © 2010, Víctor Luaña and Alberto Otero-de-la-Roza. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

HTML obfuscation

By Ben Okopnik

Whoops. Darn it, I meant to publish this article in the April issue, for obvious reasons... but between being sicker than I have been for years (I'm usually healthy as a bull), driving 700+ miles in that condition, teaching an intensive 3-day class, and then driving back - and publishing LG in the middle of all that - somehow, I managed to forget it. Perhaps perfectly appropriate for a guy with an April 1st birthday, but... well, I hope you enjoy it anyway, despite the delay.
-- Ben


There are times when you want the content of your web page to be a deep, dark secret, but you still want it to be visible. [1] Or, perhaps, you want to express your deepest, darkest feelings and yearnings in your blog, but you don't want anyone without a browser to be able to read them (take that, you browserless wimps! Oh, and you wimpettes, too.) "How," you cry, "shall this desperate need be addressed?"

Nil desperandum, my friends - it's (ta-daa!!!) Geekman to the rescue! And Perl, of course. [2]

First, you need to install the HTML::Parser module. Don't worry about the name, really; behind that deceptively innocent-sounding monicker, it's got all the proper telepathy and black-helicopter-shielding functions built in (although they're really well hidden; there's a password involved, and deadly traps in the walls, and beautiful half-naked girls in golden cages, and all that stuff, so you know that it's The Real Thing. Just like Indiana Jones and the Temple of Doom. I mean, you'd never doubt the authenticity of that, right?) The important bit is where we do all that fancy, computer-science-sounding stuff, like Object Oriented Programming, and Subclassing, and Using Methods. After all, when you give big, important, and capitalized names like that to things, they can't help but work - or at least be really, really magical!

(Next month, we'll build a time machine and a hyperspace transporter using this same technique. No, it'll be a time machine inside a hyperspace transporter - or maybe the other way around. Anyway, it'll be really exciting, trust me!)

The Code

So, OK. Here we go with the magical code:

#!/usr/bin/perl -w
# Created by Ben Okopnik on Thu Feb 27 22:00:12 PST 2003
use strict;

die "Usage: ", $0 =~ /([^\/]+)$/, " <file_to_mung.html>\n"
	unless @ARGV && -r $ARGV[0];

package HTML::Parser::Mung;
use HTML::Parser;
@UNIVERSAL::ISA = 'HTML::Parser';

sub start { print $_[4] }
sub end   { print $_[2] }
sub text  {
    $_[1] =~ s#(&(\w+);?)#${{qw/lt < gt > amp & quot "/}}{$2}||$1#ge;
	$_[1] =~ s/(\S)/sprintf "&#%s;", ord $1/eg;
	print $_[1]
}

my $p = HTML::Parser::Mung->new();
$p->parse_file($ARGV[0]);

Yay! Since it's all magical, we'll just apply it to this article. Poof! Go ahead, view the source. (It wasn't like that until you read to this point in the article. That's all part of the magic.)

Explaining the Code

Are you kidding??? Does David Copperfield explain his illusions? Does Houdini tell you how he did it? Does... well, if I told you, I'd have to kill you. The important part is, you can download this script and do it too. Be satisfied, and go away.

Otherwise, when I'm testing next month's project, I'll accidentally transport a Tyranosaurus Rex into your living room.


[1] This is in case the CIA is following you and tapping your phone, or your Web page is being targeted for assassination by Colombian drug lords, or the aliens are tapping your thoughts and the foil-lined beanie with the titanium propellor is Just Not Working. Since these things are so commonplace that all three often occur at the same time, this article seeks to redress the injustice.

[2] I mean, if you're going to either 1) perform the most amazing feats of text manipulation that will stun and amaze the most jaded audience, or 2) turn perfectly sensible text into completely senseless gibberish, there's only one possible answer. (And it's not "dadadodo" for #1 and "Microsoft Word spell-check" for #2, either.)


Talkback: Discuss this article with The Answer Gang


picture

Ben is the Editor-in-Chief for Linux Gazette and a member of The Answer Gang.

Ben was born in Moscow, Russia in 1962. He became interested in electricity at the tender age of six, promptly demonstrated it by sticking a fork into a socket and starting a fire, and has been falling down technological mineshafts ever since. He has been working with computers since the Elder Days, when they had to be built by soldering parts onto printed circuit boards and programs had to fit into 4k of memory (the recurring nightmares have almost faded, actually.)

His subsequent experiences include creating software in more than two dozen languages, network and database maintenance during the approach of a hurricane, writing articles for publications ranging from sailing magazines to technological journals, and teaching on a variety of topics ranging from Soviet weaponry and IBM hardware repair to Solaris and Linux administration, engineering, and programming. He also has the distinction of setting up the first Linux-based public access network in St. Georges, Bermuda as well as one of the first large-scale Linux-based mail servers in St. Thomas, USVI.

After a seven-year Atlantic/Caribbean cruise under sail and passages up and down the East coast of the US, he is currently anchored in northern Florida. His consulting business presents him with a variety of challenges such as teaching professional advancement courses for Sun Microsystems and providing Open Source solutions for local companies.

His current set of hobbies includes flying, yoga, martial arts, motorcycles, writing, Roman history, and mangling playing with his Ubuntu-based home network, in which he is ably assisted by his wife, son and daughter; his Palm Pilot is crammed full of alarms, many of which contain exclamation points.

He has been working with Linux since 1997, and credits it with his complete loss of interest in waging nuclear warfare on parts of the Pacific Northwest.


Copyright © 2010, Ben Okopnik. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Open Hardware and the Penguin: using Linux as an Arduino development platform

By Tom Parkin

Introduction

Linux is one of the more scalable Operating Systems in existence today, equally at home on tiny embedded single-board-computers as it is on supercomputing clusters. However there will always be some computing applications for which Linux doesn't fit; indeed for which any OS at all is something of an unnecessary overhead. Devices fitting this niche are many and varied, ranging from low-powered PIC devices to more capable microprocessors (such as those based around Intel's venerable 8051 chip). Typically each device or device family comes with its own software stack and development environment, and until recently, all the options for development at this level held one thing in common: proprietary designs, and proprietary software.

Enter the Arduino

This is beginning to change. Although hardly as vibrant and diverse as the Open Source software ecosystem, a nascent Open Source hardware movement is quietly developing open-licensed hardware designs for a wide range of applications. At the most fundamental level, the Open Cores project aims to develop Open Source hardware designs for chips themselves, ranging from memory controllers to full-blown processor designs. Moving up a layer of abstraction, projects such as The Open Graphics Project are developing hardware board designs using off-the-shelf components. Sitting somewhere inbetween the two, the Arduino project aims to provide a general-purpose microcontroller platform, integrating an Open Source board design with Open Source software development tools.

The Arduino was conceived in 2005 by academics at an Italian design school who had become frustrated with the paucity of simple, powerful platforms for their students to use as a base for projects. In classic open-source style they decided to build their own, and in a few days of creative hacking the embryonic Arduino was born. Since then it has grown from being a teaching aid to become rapid prototyping and hobbyist platform with a global community of users.

At its heart the Arduino consists of an 8-bit microcontroller from Atmel's megaAVR range of chips, which is supplemented with I/O headers and a USB or RS232 connection for uploading of code. It is a simple design which retains a shallow learning curve and provides remarkable flexibility. Properties which, in conjunction with the open-source licensing of the device's circuit design and software stack, have led to an impressive diversity of applications. Not only are Arduino hackers exploring such traditional geek territory as a Wii-nunchuk controller for a trainset, they're also developing exceedingly practical projects along the lines of a controller for a beer fridge, not to mention pushing the boundaries of computer-music creativity with a laser MIDI "harp".

Arduino development under Linux

In keeping with the Arduino's open-source heritage, all of the tools required to develop Arduino code will run on a Linux platform, and the remainder of this article will provide a tutorial for getting started with Arduino hacking.

This tutorial has been tested using a 32-bit Crunchbang Linux system, which is based on Ubuntu 9.04. Where possible the instructions have been written to be transferable to other distributions with minimal effort.

1. Obtain an Arduino board

First of all, it is necessary to source the Arduino hardware itself. Although it is possible to build an Arduino by hand using widely available components, it is best to buy a pre-built board from one of the many hobbyist websites which stock them. This avoids the need for potentially error-prone soldering or bread-board assembly at the early stages of Arduino development. Pre-built Arduino boards can be sourced from the websites linked from the Arduino project's website, and are relatively cheap. At the time of writing, the going rate is around 20 GBP or 25 USD.

2. Install the development environment

Once an Arduino board has been obtained, it is time to create a development environment on a Linux host PC. The simplest way to do this is by using the Arduino Integrated Development Environment (IDE). This is a Java application which will provide support for source code editing and compilation, binary image upload to the Arduino, and a serial communications terminal, all within a single application.

2.1 Download the Arduino IDE from the Arduino website

The latest release of the Arduino IDE at the time of writing was version 0018. It is recommended that the latest version of the IDE is used.

    # Create a new directory to store the IDE in
    mkdir arduino
    cd arduino

    # Download the IDE tarball from the web
    wget http://arduino.googlecode.com/files/arduino-0018.tgz

    # Untar the IDE tarball
    tar -xzf arduino-0018.tgz

2.2 Install the necessary supporting packages for the IDE

Since the Arduino IDE is a Java program a suitable Java Runtime Environment must be installed on a host PC. In addition, the IDE relies on some further software packages: GCC for code compilation, binutils and libc for the creation of a runnable image for the Arduino microprocessor, and avrdude for communication between the host PC and the Arduino board.

Super-user privileges will be required to install these software packages on the host PC: for the purposes of this demonstration, sudo is used.

    # Install Sun's JRE
    sudo aptitude install sun-java6-jre
   
    # Install the AVR compiler and runtime tools
    sudo aptitude install gcc-avr binutils-avr avr-libc

    # Install the AVR communications tool
    sudo aptitude install avrdude

2.3 Unprivileged user access to /dev/ttyUSB0

Contemporary Arduino boards such as the Duemilanove communicate with the host PC via a USB-serial connection. In order to access this device node as a normal user on the host PC, it may be necessary to add your user to a group with access rights to the node.

    # First, connect the Arduino board to the host PC via. a USB cable,
    # then check dmesg output on the host PC
    dmesg | grep -i "usb serial"
        [ 1054.569782] USB Serial support registered for generic
        [ 1054.569860] usbserial: USB Serial Driver core
        [ 1054.594282] USB Serial support registered for FTDI USB Serial Device
        [ 1054.594433] ftdi_sio 2-1:1.0: FTDI USB Serial Device converter detected
        [ 1054.594651] usb 2-1: FTDI USB Serial Device converter now attached to ttyUSB0

    # dmesg output suggests that the device is connected to ttyUSB0,
    # so check the access rights on that device node
    ls -l /dev/ttyUSB0
        crw-rw---- 1 root dialout 188, 0 2010-03-09 22:13 /dev/ttyUSB0

    # The group "dialout" has access to /dev/ttyUSB0, so add user "tom" to that group.
    # You should replace "tom" with your own login name, of course!
    sudo usermod -a -G dialout tom

3. Launch the IDE and connect the Arduino board

With the supporting software installed and access to the Arduino USB-serial device node configured, the IDE can be launched:

    cd arduino-0018
    ./arduino

The Arduino Duemilanove board is powered from the USB bus. All that should be required to get this up and running is to plug it into your Linux PC, via a USB cable.

4. Test your setup with an example project

The Arduino IDE is fairly intuitive and easy to use. To illustrate the basic procedures required to build and upload code, we'll use a simple demonstration project which uses the in-build LED on the Arduino Duemilanove board to flash a message in Morse code.

4.1 Example code

/*
 * morse.pde
 *
 * An Arduindo Morse code blinkenlight example for Linux Gazette
 *
 * Tom Parkin, March 2010
 * tom.parkin@gmail.com
 */

/* Digital I/O pin 13 is an LED on the Duemilanove board */
static const int ledPin = 13;

/* Encapsulates letter / code associations for Morse code */
struct {
    char    letter;
    char    *code;
} morse_letter_t;

/*
 * Morse code encoding specification, refer to wikipedia for more info:
 * http://en.wikipedia.org/wiki/Morse_code
 */
#define DOTLEN          250

#define DOT             ( pulseLed(ledPin, DOTLEN) )
#define DASH            ( pulseLed(ledPin, 3*DOTLEN) )
#define SYMBOLSPACE     ( delay(DOTLEN) ) 
#define LETTERSPACE     ( delay(3*DOTLEN) )
#define WORDSPACE       ( delay(7*DOTLEN) )

morse_letter_t alphabet[] = {
    { 'A', ".-" },
    { 'B', "-..." },
    { 'C', "-.-." },
    { 'D', "-.." },
    { 'E', "." },
    { 'F', "..-." },
    { 'G', "--." },
    { 'H', "...." },
    { 'I', ".." },
    { 'J', ".---" },
    { 'K', "-.-" },
    { 'L', ".-.." },
    { 'M', "--" },
    { 'N', "-." },
    { 'O', "---" },
    { 'P', ".--." },
    { 'Q', "--.-" },
    { 'R', ".-." },
    { 'S', "..." },
    { 'T', "-"  },
    { 'U', "..-" },
    { 'V', "...-" },
    { 'W', ".--" },
    { 'X', "-..-" },
    { 'Y', "-.--" },
    { 'Z', "--.." },
    { 0, NULL }
};

/* The setup function for the Arduino */
void setup()
{
    pinMode(ledPin, OUTPUT);
}

/* Turn the LED attached to 'pin' on for 'milliseconds' ms */
void pulseLed(int pin, unsigned int milliseconds)
{
    digitalWrite(pin, HIGH);
    delay(milliseconds);
    digitalWrite(pin, LOW);
}

/* Flash 'letter' in Morse on the LED, based on 'alphabet' lookup */
bool encodeLetter(char letter, morse_letter_t *alphabet)
{
    int i;
    char *code = NULL;

    /* Lookup the specified letter in the alphabet */
    for (i=0; alphabet[i].letter != 0; i++) {
        if (alphabet[i].letter == letter) {
            code = alphabet[i].code;
        }
    }
    
    if (!code) return false;

    /* Display the letter */
    for (i=0; i < strlen(code); i++) {
        if (code[i] == '.') {
            DOT;
        } else {
            DASH;
        }
        SYMBOLSPACE;
    }
    return true;
}

/* Flash 'word' in Morse on LED, based on 'alphabet' lookup */
bool encodeWord(char *word, morse_letter_t *alphabet)
{
    int i;

    for (i=0; i < strlen(word); i++) {
        encodeLetter(word[i], alphabet);
        LETTERSPACE;
    }
    return true;
}

/* Arduino 'main' loop */
void loop()
{
    delay(5000);
    encodeWord("LINUX", alphabet);
    WORDSPACE;
    encodeWord("GAZETTE", alphabet);
    WORDSPACE;
}

There are a couple of points to note about this simple Arduino "sketch" (n.b. a "sketch" is the Arduino term for a project). Firstly, although it looks like C code, perceptive readers will have noticed that it isn't actually a complete program. That's because the Arduino IDE provides a lot of basic functionality behind the scenes, leaving the programmer to focus on the task at hand rather than the intricacies of driving a given I/O pin high, for example.

A full reference to the Arduino environment is available via the Arduino website. Essentially it boils down to a library of utility functions (for example delay(), which delays program execution for the specified number of milliseconds), and two callbacks which the core Arduino code calls to allow a given sketch to do its work. As shown above, these callbacks are setup() for initial configuration, and loop() for the main event loop of the program.

4.2 Create a new Arduino sketch

To import this code into the Arduino IDE, launch the IDE then click File -> New to create a new sketch. Copy and paste the code into the editor window, then save it (File -> Save) with a sensible filename.

4.3 Configuring the IDE

Now configure the IDE to talk to your Arduino board.

First, instruct the IDE to use the serial port that your Arduino is attached to by clicking Tools -> Serial Port and selecting the serial port that we determined previously (this was /dev/ttyUSB0, in our example). Depending on your distribution you may need the Arduino to be connected for this port to show up.

With that done, set the IDE to compile code for your specific Arduino device by clicking Tools -> Board. In your author's case the correct choice is Arduino Duemilanove or Nano w/ATMega328, which is the most recent Arduino board at the time of writing.

4.4 Compile and run: the moment of truth

Finally, to compile and upload the code to the Arduino, click Sketch -> Verify/Compile, and then File -> Upload to I/O board. Simple!

If everything has gone according to plan, the Arduino IDE should load the compiled morse code application to the board. Look for the TX and RX lights flashing on the board to indicate communications in progress. The board should immediately run the program. After a few seconds, the Arduino should start blinking its green LED to flash a Morse code message.

What next?

At this stage, you should be well on your way to getting to grips with the fundamentals of Arduino development, and hopefully you'll already be envisaging how you could use the platform in an exciting new project.

It's not working!

As with any computer-related activity, Arduino development has its fair share of pitfalls and sticking points to negotiate. Although the IDE and the example code above provide a simple introduction, once you set out on your own you'll inevitably hit a problem at some point. Happily the Arduino project provides many avenues for support, including the Arduino forums and IRC channel. In addition, the Arduino and its IDE have been widely blogged about, and a small amount of work with a search engine will yield a lead on most of the widely-encountered issues.

What if I don't want to use the IDE?

Although the Arduino IDE is a great tool, some developers may wish to use their favourite editor for generating code instead. This is made easily possible using Kimmo Kulovesi's excellent arduino_make.sh script, which leverages the internal build files used by the IDE to perform the basic Arduino compilation and upload processes from the commandline.

Conclusion

The Open Source Hardware movement is increasingly demonstrating that the power of open development is not limited to the software domain. The Arduino project leverages the best of both Open Source hardware and software to deliver a great multi-purpose microcontroller platform. Whether you're a software hacker looking for an easy entry into the world of embedded devices, a teacher seeking a device to base an electronics course around, or a seasoned hardware designer looking for a quick prototyping environment, you owe it to yourself to give the Arduino a look.


[ For an amusing bit of synchronicity this month, take a careful look at our XKCD comic strip. Arduino is the hot new thing! :) -- Ben ]


Talkback: Discuss this article with The Answer Gang


[BIO]

Tom Parkin has been fascinated by the inner workings of digital technologies ever since his father brought home a VIC-20 sometime in the mid-eighties. Having spent most of his childhood breaking computers in a variety of inventive ways he decided to learn how to fix them again, a motivation which lead him to undertake an MEng degree in Electronic Systems Engineering in 2000. Since graduating he has pursued a career in embedded software engineering, and now feels that he has probably been responsible for more working computers than broken ones.

Tom was introduced to Linux when a friend lent him a thick stack of Mandriva installation CDs, and he has been using Open Source software ever since. Like most Linux users, Tom has tried many different distributions but is currently settled with Fedora at work and Crunchbang on his home machine.

When not tinkering with computers and Linux, Tom enjoys exploring the great outdoors on bike or on foot, and making music.


Copyright © 2010, Tom Parkin. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Hey, Researcher! Open Source Tools For You!

By Amit Kumar Saha

Disclaimer:
Your author is writing this article after having just finished his latest task - conducting numerous experiments which involved processing a lot of data, and finally writing the paper on it in just under five days. He wonders how he would do it without some of the tools he will be talking about in this article, leaving aside the pleasure of working on Linux systems. His views are biased and are meant to be taken cum grano salis, but I hope you will give him a fair chance.

What am I addressing in this article?

My meeting with Linux happened accidentally 10 years ago, with the fun of wiping out and putting in a Linux distribution numerous times on my first computer without having any clue about how to work on it. Finally, after a lot of such iterations I finally settled in, working on it full time for the last 5 years. The accident that happened 10 years ago stood me in good stead both when I was working at a software company and now when I'm working in a research lab. In a research lab, we need to plot figures, run batch jobs remotely, perform scientific and numerical computing, and last but not least, publish results by writing papers. Just a few days before I wrote to the Editor-in-Chief about this article idea, I helped a Windows friend draw some plots (fixing them for publication) for her paper, wondering all the while "Jeez, how do Windows users do it?" Most tools that I used were primarily for Linux. On Linux, things are so much easier. You can focus on the real job instead of bothering about setting up your tools. This article is therefore an unabashed claim about the coolness of working with these tools while you focus on your research. Some of these tools can be set up on a non-Linux system too; however, I shall address their usage on a Linux system only and make no attempt to cover any non-Linux issues. The primary focus here is to introduce my readers to these tools, and not to convert them to Linux. It would, in fact, be a good exercise to get all these tools that I talk about here on Windows!

In this article, I shall take a look at (in no particular order) getting up and on with LaTex, gnuplot, GNU octave, Python scientific libraries, Beamer, and some other tools and libraries.

Hello Linux

Before I start talking about using the scientific tools, I shall write a few lines about how you can set up a Linux system for yourself - if you don’t already have one, that is. While bare-metal Linux installations are the best case scenario, it may not always be possible because your primary work may be on Windows. In such cases, the best solution is to go for a virtual Linux installation. In my humble opinion, I have found using VirtualBox to be a breeze when you need to set up a virtual machine. Just download a Linux distribution like Ubuntu and install it using VirtualBox. Once you have the basic installation, take a little time to gain some hands on familiarity with the terminal, using the package manager to install packages, and using a text editor.

Here are some resources which can help you getting started:

Giving LaTex a try

LaTex is a document preparation system for high-quality typesetting and is widely used for writing scientific papers, technical manuals, and even creating presentations (more on this later). TeX Live is an easy way to get up and running with LaTex on your Linux system. A file written in LaTex is usually saved as a .tex file. Once you have created your .tex file, you will compile the file using the latex command, which gives you a dvi file, which you can convert to a PDF file using a tool like dvipdf. You can also directly get a PDF output using the tool pdflatex.

You can edit LaTex in your text editor of choice. The widely used VIM and Emacs both support LaTex editing with syntax-highlighting, code-completion, etc.

The best way to get started with LaTex is by getting started. Start your text editor, type in some Latex and see the results. Here are some great resources to help you master it:

gnuplot

If you are doing experimental work, there is a fair chance that you are going to present your results in a graphical plot. Please welcome gnuplot. It is a command-line driven graphing utility for creating 2-dimensional as well as 3-dimensional graphical plots. It supports output to many file formats with eps and fig being of special interest to the researcher. The easiest way to install gnuplot is using the package manager of you Linux distribution.

Here are a couple of links to help you get started with gnuplot:

Calling gnuplot from your C program

A cool way to use gnuplot is to use it to display continuously changing data that your program may be generating. For this example, let’s assume that you have a program written in the C programming language and you want to send the data it's generating to gnuplot. The simplest way to do this is to use the popen system call. Please refer to this 2-cent tip published here sometime back.

There is also an ANSI C interface to gnuplot available here, but I haven’t personally used it.

GNU Octave

GNU Octave (referred to as Octave from now on) is a software tool intended for numerical computation. Octave has extensive tools for solving common numerical linear algebra problems, finding the roots of nonlinear equations, integrating ordinary functions, manipulating polynomials, and integrating ordinary differential and differential-algebraic equations. Octave is available for use on Linux, Solaris, Mac OSX, and Windows. If your study or play involves numerical computations, Octave is for you. Octave has been designed with MATLAB compatibility in mind, so with careful design you could write scripts which run on both MATLAB and Octave. It is easily extensible and customizable via user-defined functions written in Octave’s own language, or using dynamically loaded modules written in C++, C, Fortran, or other languages.

Octave is most definitely available in the package repository of your Linux distribution and that is the easiest way to get it.

GNU Octave comes with extensive documentation, which is the best place to get started.

In case you happen to like this author’s writings, he is currently writing an article series on Octave elsewhere. Please refer to these blog post for PDFs of the articles.

Add Python to your toolkit

I shall not use this section of the article for Python language advocacy. Rather, I shall point you to some ways you may consider using Python in your research, if you are already using it, or think of using it in the future as just another of your many experiments. Python is already installed on most of the Linux distributions. Let us now see some of the ways you can use Python:

As a scientific calculator

With its math module, Python can be a very useful command-line scientific calculator. For example:

$ python
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import math
>>> math.factorial(10)
3628800
>>>

As you can guess, you will have to import the math module every time you want to use it. You can avoid doing this by automatically importing it at startup:

  1. Create a file: .pythonrc in your $HOME and place this line:
    import math
    
  2. Now in your .bashrc or similar:
    export PYTHONSTARTUP=$HOME/.pythonrc
    

Now, every time you start Python interactively, you should have the math module already imported.

$ python
Python 2.6.4rc1 (r264rc1:75270, Oct 10 2009, 02:40:56)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> math.pi
3.1415926535897931

The decimal module is another interesting module which provides support for decimal fixed point and floating point arithmetic. fractions is another interesting module, which can be used for rational number arithmetic.

The beauty is that you do not need to know the Python programming language to use these modules. Just fire up the Python interpreter and import the module(s) you want to use, then call the appropriate function.

Scientific libraries

If you are already conversant with Python, then you would definitely like to know about some scientific libraries which you may find useful in your research:

Beam it with Beamer!

You have done your experiments, sent in your findings, they've been accepted, and now you are going to present them at a conference. You want to prepare some slides to give you company as you talk about your work. OpenOffice.org Impress, KOffice? How about taking a look at something different? You will surely like the results, I promise. Beamer is a LaTex class for making presentation slides. In LaTex parlance, it is just another Document Class. You shouldn’t even make an attempt to use Beamer, if you have no idea what LaTex is. However, if you already use LaTex, chances are you swear by it, then Beamer is for you.

The easiest way to install is to use the package manager in your Linux distribution. You will see that some more packages providing other extra LaTex classes need to be installed. Prominent among these are ’latex-xcolor’ and ’pgf’.

There is, of course, a way to install the LaTex class manually. It's easy to do it, but not easier than the previous method. The LaTex Beamer user guide shows how to do it.

With the danger of being pulled up by the chief editor for self-publicity, the author would like to point to his article Typesetting Presentations with Beamer, published elsewhere. The PDF copy is available from here.

Others you may find useful

In this section, I shall talk about some of the other tools, which are useful for scientific computing that I know of:

The small ones

One category of tools that I haven't talked about yet is some of the Linux utilities, the ones that ship along with the Linux system - Shell scripting, awk and sed. My knowledge of each is very limited, but I have a working knowledge of them all, and I can get things done with them, which is what matters a lot of the time.

Conclusion

It's often the case that we do not know of a great tool which would simplify our work as we go about our research work everyday. My hope as a novice researcher is to acquaint the reader with the kind of software tools available in the Open Source world. I have made no attempt to be exhaustive here. There are a lot of other tools out there, which I haven’t yet used, so I have refrained from writing about them. I hope you have enjoyed reading this article as much as I have writing it!

Acknowledgements

Credits are due to the folks on the mailing lists and the documentation of all the projects that I have talked about today, because they are the source of my learning.


This document was translated from LATEX by HEVEA.

Talkback: Discuss this article with The Answer Gang


Bio picture

The author is a part-time freelance technical writer from India and he writes on all things Linux and Open-Source-y.


Copyright © 2010, Amit Kumar Saha. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

A quick look at what's coming with Fedora 13

By Anderson Silva

It's Wednesday morning, and it's been exactly 24 hours since the Fedora Project released the beta for Fedora Linux 13th version. If you are not too familiar with Fedora's release schedule, let me give you a quick overview. The Fedora Project tries to release a new version of Fedora around May 1st and October 31st every year.  Obviously as with most software projects these dates are goals, and not deadline. For example, Fedora 13, codenamed Goddard, is scheduled to be released on May 18th.

Usually a month or so before the final release of the latest Fedora version, there is a release of a beta, and that's what I am running right now. Like I said, the beta has only been out for 24 hours, and there is no way I've had the chance to go over every single new feature that's coming out with Goddard, but I've been able to verify a few, and those are the ones I will share with you.

Live "DVD" Installation

That's right, DVD. At least for this beta distribution Fedora Project has made available a 900+ MB iso for the live image of the OS. Their site does note that for the final release in May, the image will be back down to fit on a CD. The main reason for this change is the inclusion of OpenOffice to the live image. It's always worth reminding everyone that these live images can also be written to USB drives non-destructively; that is, you can make the disk bootable run and install Fedora live without reformatting your USB drive. For more information see: http://fedoraproject.org/wiki/FedoraLiveCD/USBHowTo

screenshot
Screenshot 1: A new window in the Fedora live installer that allows the user to pick how they want the OS to be installed on the hard drive.

Printing

Remember that scene from 'Office Space' where Peter, Samir and Michael take the office printer to some isolated area and beat the daylights out of it? We've all had that same type of feeling towards printers.  I'd even go as far as saying that Linux users may have suffered even more for lack of open source compatible drivers throughout the years. The problem hasn't been completely solved, but we are getting there one release at a time. Fedora 13 will introduce an easier automatic way for your printer drivers to get installed. It detects the printer available, and suggests which RPM needs to be installed to get your system working.

screenshot
Screenshot 2: My HP Photosmart was automatically detected on the network when trying to add a printer.

screenshot
Screenshot 3: Fedora suggest the driver needed to get my printing needs met.

screenshot
Screenshot 4: I am ready to print

Network Manager Command Line Interface

One of the most user-friendly technologies to pop up around the Linux desktop in the past few years is arguably NetworkManager. It has made detecting network hardware, wifi SSIDs, and broadband cards as easy as plug it in, click, and you are connected. However, this friendliness did not come without a cost. For system administrators like me, the preference of a command line interface for NetworkManager made it virtually impossible to be used in server-based environments.

Fedora 13 will be trying to fill this gap by introducing nmcli, a command line utility that will allow users who prefer the command line to interact with NetworkManager. As of yesterday's Beta's release, this utility still doesn't have a main page, but I am trusting the Fedora community to have one up by the time the final release is out.

screenshot
Screenshot 5: A couple of examples of nmcli where it shows the status of devices available on my laptop and the connections they are using.

screenshot
Screenshot 6: nmcli -p dev wifi will show the SSIDs available for you to connect to.

I can tell NetworkManager to connect to my default SSID by running a command such as this:

[ansilva@caroica ~]$ nmcli -p con up uuid eceac525-cd30-4df0-8631-73045b42e16e
Active connection state: activating
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/

Languages:

I am originally from Brazil, and even though I don't think I've ever needed to change my language setting in Fedora, I do know enough people in Brazil, especially new Linux users, who find being able to switch languages in the distribution a nice perk. Fedora 13 will bring a yum plugin that automatically searches for the language RPMs when not already installed on the system.

screenshot
Screenshot 7: Picking Portuguese as the default language will prompt yum to download the correct packages.

screenshot
Screenshot 8: Automatic download. Once installed, just log out and log back in to see the changes.

iPhone, iPod Touch Support

I have to be honest, I was skeptical when it was announced that Fedora 13 would work with iPhones and iPod Touches, but I tried it and it worked! I typed in my iPhone passcode, plugged into the USB port, and Fedora recognized my device as a drive, as a camera and as a mp3 player. Very impressive!

screenshot
Screenshot 9: My iPhone mounted on Fedora 13 showing my playlists under Rhythmbox.

If you need multimedia codecs to play your media, you can legally purchase them at: http://www.fluendo.com/. They even have DVD playback software for Linux, including Fedora.

What else is new?

screenshot
Screenshot 10: Fedora 13 comes with Gnome 2.30, for KDE users, 4.4 is the version packaged for the distro.

There are plenty of other new features and changes in Fedora 13.  For example, NFSv4 will become the default version with support for IPv6, and installations and upgrades can now be performed over the internet with boot images available at:  http://boot.fedoraproject.org/.

Even though it started in Fedora 12, it's worth mentioning that Gimp is not a part of the default live image any more, and as of now Fedora 13 Shotwell Photo Manager is. A new GUI tool, called deja-dup,  for backup and restore of the system is now part of the OS, although backintime is still my tool of choice.

Conclusion

During the next few days, or even weeks, there will be plenty of blog posts, press releases and articles written about 'what's new' on Fedora 13.  My wish was to give you a quick first-hand look at the changes that most users will start noticing right from the installation process. You can take a more in-depth look at the major features coming out with Fedora 13 at: https://fedoraproject.org/wiki/Releases/13/FeatureList and read its Release Notes draft. Be aware that we are dealing with a beta version here, so expect issues and problems. I am not even recommending that you install this beta version, but, if you do, keep an eye on Fedora 13's Common Bugs page for workarounds and known issues.

And finally, if you do find a bug yourself and would like to report it, you can do so by logging in to https://bugzilla.redhat.com/.


Talkback: Discuss this article with The Answer Gang


[BIO]

Anderson Silva works as an IT Release Engineer at Red Hat, Inc. He holds a BS in Computer Science from Liberty University, a MS in Information Systems from the University of Maine. He is a Red Hat Certified Engineer working towards becoming a Red Hat Certified Architect and has authored several Linux based articles for publications like: Linux Gazette, Revista do Linux, and Red Hat Magazine. Anderson has been married to his High School sweetheart, Joanna (who helps him edit his articles before submission), for 11 years, and has 3 kids. When he is not working or writing, he enjoys photography, spending time with his family, road cycling, watching Formula 1 and Indycar races, and taking his boys karting,


Copyright © 2010, Anderson Silva. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Password Management with KeePassX

By Neil Youngman

Introduction

Many of our readers will be familiar with the problems of password management. How do you keep track of logins for dozens or hundreds of systems and websites without reusing the same user name and password over and over? How do you generate passwords for different sites with different constraints on what constitutes an acceptable password? How can you securely transfer password lists between systems?

There are a whole bunch of solutions to some of these requirements: I use kwallet, Firefox's password manager and Opera's wand to meet some of my requirements, but they all have serious limitations. To meet all those requirements I now use KeePassX, which is a free (GPL), cross platform password safe. It also incorporates a password generator. KeePassX is easy to use and employs high grade encryption to keep your passwords safe.

This article will give a brief introduction to KeePassX and demonstrate how to use it.

Simple use as a password safe

Figure 1

When you open KeePassX for the first time you get a window with a number of blank panes. Passwords can not be saved at this point. All passwords must be stored in a group, within a database.

Start by selecting "File"->"New database" (shortcut CTRL-N). or clicking the left-most icon on the tool bar. You will be prompted for a password (or a key file):

Figure 2

The button to the right of the password field toggles password visibility, so you can read the password and check that it is correctly typed, or hide it from other people in the room.

After creating a database the next step is to create a group with "Edit"->"Add New Group" (shortcut CTRL-G). You need to enter a group name and you can also select an icon to go with the group name, if you don't want the default key icon. I have created a group called, surprisingly, "Example" for this article.

Once a group has been created passwords can be entered with "Edit"->"Add New Entry" (shortcut CTRL-Y). This pops up a dialog allowing entry of user names, passwords and a selection of other details.

Figure 3

The entry can include a title, a user name, a URL for web pages, an expiry date and a comment. If this is not enough, you can also attach arbitrary files. If you need to see what you are typing the button with the eye icon to the right of the password toggles password visibility. When entering the repeat password the background is red until the repeat password matches the main password.

Having entered your passwords In KeePassX, they are easy to access in a secure fashion. By default passwords are not displayed, but this can be toggled in the view menu. They can be used without having to display them, by right clicking on the entry and selecting "Copy Password to Clipboard". They can then be pasted where they are needed in the usual way.

Using the password generator

There are 3 ways to access the password generator. To simply generate a password select "Extras"->"Password Generator" or use CTRL-P as a shortcut. If you are generating a password for an entry in the database, the password generator can also be accessed via a button labelled "Gen." next to the "Password Repet.:" entry box.

Figure 4

The password entry screen allows the customisation of password generation according to various criteria, such as length and whether or not to use upper and lower case, numbers, spaces, special characters, minus and underscore. There are also tabs for generating "pronounceable" passwords and using custom sets of characters. The custom tab does not offer an obvious way of entering ranges of characters and a quick scan of the source code suggests that this is not a supported feature.

When you have selected your criteria, press the "Generate" button to create a password. If you are not saving the password, it seems that you have to make the password visible, by pressing the eye icon, to copy it. If you are editing an entry, clicking "OK" in the generate dialog fills in the password.

Cross Platform Use

The KeePassX downloads page offers source code plus packages for Windows, MacOS X, Ubuntu, OpenSuSE and Fedora. KeePassX is also in the official Debian repositories.

Password databases generated on Linux or MacOS can be copied to Windows PCs and accessed from there and vice-versa.

Other Features

KeePassX offers a choice of AES(Rijndael) or Twofish encryption algorithms in "File"->"Database Settings". The number of encryption rounds can also be adjusted.

KeePassX can import PwManager files and XML files exported from kwalletmanager. It can export to plain text or XML.

Postscript

In between writing this article and it's publication, I was notified that a website on which I have a login had been cracked and I should consider the user name and password compromised. This password dated back to before I started using KeePassX and like many of my older logins, it had not been updated to use stronger passwords. At that time I had also followed a practice of using a limited number of user names and very simple passwords on "low value" sites to minimise the amount I had to remember. As a consequence a number of logins were at risk if the attackers tried that user name and password on other sites I used.

I was lucky that the user name on this site was not the user name I used most often, so the number of sites at direct risk was relatively small. This has, nevertheless, provided the motivation for me to update my password on all "low value" logins and any other passwords that appear weak.

In the process of updating these passwords I have noted that the "password strength" measure provided by KeePassX is essentially useless as it is calculated by simply multiplying the password length by 8. The string "aaaaaaaa" is given the same strength as "hr~9kl_p7". I would prefer no strength indicator to a misleading indicator. [1]

I have also noted that when you copy a password from KeePassX, all the passwords are erased from klipper (KDE's clipboard history tool) a short time thereafter.

[1] http://en.wikipedia.org/wiki/Password_strength#Entropy.2C_or_bit_strength


Talkback: Discuss this article with The Answer Gang


Bio picture

Neil is a programmer, specialising in C++ on Unix and Linux. He has degrees in Computer science and Next Generation Computing.

Neil has worked on a wide range of systems from the control system for the British Gas national grid to video servers for the Home Choice video on demand service. He first programmed computers in 1980 with his school General Studies class, which was allowed access to a mainframe at The National Institute of Oceanography, programmed in Fortran on punch cards.

A computer science degree followed at Queen Mary College, London, then Neil worked for Logica for 3 years before taking an MSc in New Generation Computing at Exeter University.

The next 5 years saw Neil researching parallel simulation algorithms at the Royal Signals and Radar Establishment, initially on transputers and subsequently on SPARC based parallel systems. Since leaving RSRE, Neil has mostly worked freelance and has worked on financial data feeds, video servers and virus scanning proxies.

Neil first used Unix at college in 1982 and started working on Linux in 1996.

As of May 2004, Neil is working for Wirefast a global messaging company.

Outside of computing, Neil is into motor sport, particularly Formula 1, the World Rally Championship and the British Touring Car Championship. He doesn't race himself. If you've seen Neil's driving, you'll understand why.

Copyright © 2010, Neil Youngman. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

HelpDex

By Shane Collinge

These images are scaled down to minimize horizontal scrolling.

Flash problems?

Click here to see the full-sized image

Click here to see the full-sized image

All HelpDex cartoons are at Shane's web site, www.shanecollinge.com.

Talkback: Discuss this article with The Answer Gang


Bio picture Part computer programmer, part cartoonist, part Mars Bar. At night, he runs around in his brightly-coloured underwear fighting criminals. During the day... well, he just runs around in his brightly-coloured underwear. He eats when he's hungry and sleeps when he's sleepy.

Copyright © 2010, Shane Collinge. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

XKCD

By Randall Munroe

[cartoon]
Click here to see the full-sized image
[cartoon]
Click here to see the full-sized image
More XKCD cartoons can be found here.

Talkback: Discuss this article with The Answer Gang


[BIO]

I'm just this guy, you know? I'm a CNU graduate with a degree in physics. Before starting xkcd, I worked on robots at NASA's Langley Research Center in Virginia. As of June 2007 I live in Massachusetts. In my spare time I climb things, open strange doors, and go to goth clubs dressed as a frat guy so I can stand around and look terribly uncomfortable. At frat parties I do the same thing, but the other way around.


Copyright © 2010, Randall Munroe. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

Doomed to Obscurity

By Pete Trbovich

These images are scaled down to minimize horizontal scrolling.


Click here to see the full-sized image


Click here to see the full-sized image


Click here to see the full-sized image

All "Doomed to Obscurity" cartoons are at Pete Trbovich's site, http://penguinpetes.com/Doomed_to_Obscurity/.

Talkback: Discuss this article with The Answer Gang


[BIO]

Born September 22, 1969, in Gardena, California, "Penguin" Pete Trbovich today resides in Iowa with his wife and children. Having worked various jobs in engineering-related fields, he has since "retired" from corporate life to start his second career. Currently he works as a freelance writer, graphics artist, and coder over the Internet. He describes this work as, "I sit at home and type, and checks mysteriously arrive in the mail."

He discovered Linux in 1998 - his first distro was Red Hat 5.0 - and has had very little time for other operating systems since. Starting out with his freelance business, he toyed with other blogs and websites until finally getting his own domain penguinpetes.com started in March of 2006, with a blog whose first post stated his motto: "If it isn't fun for me to write, it won't be fun to read."

The webcomic Doomed to Obscurity was launched New Year's Day, 2009, as a "New Year's surprise". He has since rigorously stuck to a posting schedule of "every odd-numbered calendar day", which allows him to keep a steady pace without tiring. The tagline for the webcomic states that it "gives the geek culture just what it deserves." But is it skewering everybody but the geek culture, or lampooning geek culture itself, or doing both by turns?


Copyright © 2010, Pete Trbovich. Released under the Open Publication License unless otherwise noted in the body of the article. Linux Gazette is not produced, sponsored, or endorsed by its prior host, SSC, Inc.

Published in Issue 174 of Linux Gazette, May 2010

The Linux Launderette

Google Annotations Gallery

Jimmy O'Regan [joregan at gmail.com]


Thu, 1 Apr 2010 11:16:19 +0100

In keeping with the spirit of the day, Google have released a set of open source annotations for Java: http://google-opensource.blogspot.com/2010/03/google-annotations-gallery.html (project: http://code.google.com/p/gag/)

This adds some missing features to Java:

    @ThisWouldBeOneLineIn(language="C++",
toWit="vwords[ivwords]->add_ignored_string(str);")
    private void vwords_set_superficial(int ivwords, String str) {
        TaggerWord tmp = new TaggerWord();
        tmp = vwords.get(ivwords);
        tmp.set_superficial_form(str);
        vwords.set(ivwords, tmp);
    }

It also includes @LOL, @Facepalm, @WTF, and @AnimalsHarmedDuringTheMaking

-- 
<spectie>       Jim only looks at them through the reflection in a mirror
<Aha_>  he told me there are 'some' errors
<spectie>       hahaha
<spectie>       he was being extremely generous
<spectie>       it is more correct to say that "amazingly, there is
some stuff that isn't wrong"

[ Thread continues here (4 messages/6.40kB) ]


The DBAD licence :)

Predrag Ivanovic [predivan at nadlanu.com]


Thu, 15 Apr 2010 18:31:52 +0200

Author: Stephen Eley

"This is a proposed draft of the Don't Be a Dick license for open source projects. The purpose of this license is to permit the broadest feasible scope for reuse and modification of creative work, restricted only by the requirement that one is not a dick about it."

Full text at http://github.com/SFEley/candy/blob/master/LICENSE.markdown

Awesome :) Pedja

-- 
 Give a man a match, and he'll be warm for a minute, but
 set him on fire, and he'll be warm for the rest of his life.

[ Thread continues here (5 messages/7.32kB) ]


Anyone using Google Store Anything yet?

Deividson Okopnik [deivid.okop at gmail.com]


Thu, 1 Apr 2010 13:33:06 -0300

Heres some info on the features:

http://googleenterprise.blogspot.com/201[...]-and-store-anything-in.html#comment-form

:P

[ Thread continues here (6 messages/8.20kB) ]



Talkback: Discuss this article with The Answer Gang

Published in Issue 174 of Linux Gazette, May 2010

Tux