Reply to topic  [ 2 posts ] 
reding/writing files with many lines 
Author Message

Joined: Fri Dec 07, 2012 12:12 am
Posts: 1
Post reding/writing files with many lines

I'm trying to write a file with more than 1 million lines using Yorick. For some reason it only writes up to 1048576. I've tried writing line by line in a loop, but the result is the same.

If I write separate files (1 million lines each) and then merge them using the 'cat' command in Unix terminal, when I ask Yorick to open and read the file (with rdfile(open("merged_file")) ), it only reads until 1048576, but the merged file has the correct number of lines.

Is there something I can change in Yorick so it reads/writes the number of lines I need without having to merge several separate files into one?


Thu Jan 10, 2013 5:56 am
Yorick Master

Joined: Wed Jun 01, 2005 11:34 am
Posts: 112
Post Re: reding/writing files with many lines
I notice that 1048576 == 2^20, so that sounds like you're hitting an allocated buffer.

I tried writing out more than 1 million lines and that worked for me:
> foo = indgen(2^21)
> f = open("test.txt", "w")
> write, f, format="%d\n", foo
> close, f

Examination of the file revealed that it went all the way to line 2097152.

As for rdfile, if you do "help, rdfile", it tells you that it has a second optional argument "nmax" which defaults to 2^20. So the behavior you're getting there is expected. If you know your file will have more than that many lines, specify a higher nmax.

Wed Jan 16, 2013 1:00 pm
Display posts from previous:  Sort by  
Reply to topic   [ 2 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.