Reply to topic  [ 1 post ] 
clog files no longer made by default 
Author Message
Yorick Master

Joined: Mon Nov 22, 2004 9:43 am
Posts: 354
Location: Livermore, CA, USA
Post clog files no longer made by default
I have changed the createb and updateb functions so that they do not create contents log (clog) files by default any more. You can supply a clog=1 keyword if you want the old behavior. Since the clog file is destroyed when the file closes, this should be a transparent change for most users.

The reason for the change is that on peta-scale computers, we create binary files by the millions, and with one clog file per binary file, which dribbles output a few bytes at a time, those little files turn out to dominate the file I/O time in a way that just doesn't happen with tiny numbers of files (say a few thousand). Originally, the clog files were for error recovery, but the new filesystems are so unreliable, and the failure modes are so egregious, that the clog files simply serve no useful purpose any longer.

There will soon be a related change that gives you more control over buffering of binary files, aimed again at the gigantic parallel filesystems, which need huge buffers to get anywhere close to their rated performance.


Mon May 10, 2010 10:09 pm
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 1 post ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.