LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

How to cut big text file into several small text files?

Solved!
Go to solution

Hi All,

 

 

I got a big text file.  Notepad couldn't open it.  I want to cut it into several small text files so that I can open and check it.  I use "set file position" and "set file size", but it always gives me the part of the file from begining.  I want it to, say, four small files, the first quarter, the sencond quarter, the third quarter and the fourth quarter to each file.

 

How? Seems very simple question. I must miss ...

 

 

Thanks a lot for any help.

 

Anne

Message 1 of 27
(29,056 Views)

Anne:

 

My first thought is, "WOW!  Why would you want to look at a file that big?"  I think I'd go nuts trying to absorb all that information.  😮  🙂  I'm guessing this is a bunch of data.  Try opening it up in MS Excel as a delimited file.  At least it will be formatted in a sane way.  Hopefully.  😉

 

Bill

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 2 of 27
(29,046 Views)

I really need to open and check it. It's from the site. Something goes crazy and I need to find out what happened.

 

Actually it is a log file, not very much data there.

 

The problem is the file is too big.  Notepad refuses to open it.  Wordpad opens it, but very hard to check it since it takes too much memory.

 

 

Please help.  😞

 

Anne

0 Kudos
Message 3 of 27
(29,041 Views)
Solution
Accepted by topic author Anne Zuo

Hi Anne,

 

just read blocks of the file in a loop...

readtxt.png

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
Message 4 of 27
(29,039 Views)

Well, if notepad wont open it, excel sure wont. Excel has a row limit of having 256 Columns and 65536 rows so he is talking about more than that. As I am sure you have discovered, dealing with files that big brings out the short comings in your machine, hardware, OS, and software.

 

Your options depend on the format of your data. I would not suggest splitting your data file into a set amount of files. This is because if you have a really huge file, quarters or halfs may be too big still. What you really want is a set file size that the splits will be. This insures that each file produced from the original file is manageable.

 

Tab Delimited Data

 If you have tab delimited data like what the "write to spreadsheet file.VI" produces, you can use the "read from spreadsheet file VI". It has an option that lets you choose how many rows you want to read from an offset. So you could write a loop that writes a certain number of rows and reads the offset from a shift register to a new file, initialized at 0. It will iterate through the file until it hits the end of file mark. 

 

Other Text Files

If your text is not in any particular format, things get fun. In labview the only way I have found to do what you are trying to do is to use the read from text file VI. With this VI you tell it how many characters you want to read. You would do the same loop procedure I described above creating new files of a set size and iterating through them. In order to accomplesh the offset that you need you must use the "Set File Position.vi"

 

If you want the resulting files to be rather large, DO NOT just read a huge chunk of data and write it directly to file. You will watch your system crash or seriously bog down if you try to do this. You want to create an embedded loop where the outer loop is creating the new files and the inner loop is reading smaller chunks and writing them to that file a set amount of times until you hit your desired file size for each new file.

Message Edited by rex1030 on 08-28-2009 02:01 PM
---------------------------------
[will work for kudos]
Message 5 of 27
(29,033 Views)

Thanks a lot for your reply.

 

I did it by reading the big file block by block.  A little bit painful, but works.

 

 

Cheers.

 

Anne

0 Kudos
Message 6 of 27
(29,014 Views)

Thanks. I did it. You got Kudos.

 

Cheers.

 

Anne

Message 7 of 27
(29,014 Views)

Hello,

 

I had a problem with a txt file with data in it and Labview returned the error "Labview out of memory message: memory full; not enough memory to complete this operation" Don't panic I was very unhappy when this appeared on my screen . The solution is to feed labview or any other programming language smaller files and a program called HJsplit is handy in this process.It converts the files into the file name + 001 , 002 , 003 00n and so on and so on. I have been looking at this problem for a day and found this solution to be the best help.

 

Then set up your LV program to read the files in using the Recursive File List VI and your away Smiley Happy

 

Using labview since LV 8.6 , Great Stuff 🙂
0 Kudos
Message 8 of 27
(28,525 Views)

Forgot the LINK Smiley Very Happy

 

http://hjsplit.en.softonic.com/

 

Happy , EASY file splitting Smiley Wink

Using labview since LV 8.6 , Great Stuff 🙂
0 Kudos
Message 9 of 27
(28,517 Views)

Hi Robert,

 

my 2 cents:

- using a FileSplitter, which is not aware of the data structure, may damage your data by splitting data structures

- what is wrong in using LabVIEW to read a big file block-wise?

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
Message 10 of 27
(28,496 Views)