node.js modify file data stream?

Question

I need to copy one large data file to another destination with some modifications. fs.readFile and fs.writeFile are very slow. I need to read line by line, modify and write to new file. I found something like this:

fs.stat(sourceFile, function(err, stat){
    var filesize = stat.size;

    var readStream = fs.createReadStream(sourceFile);

    // HERE I want do some modifications with bytes

    readStream.pipe(fs.createWriteStream(destFile));
})

But how to make modifications ? I tried to get data with data event

readStream.on('data', function(buffer){
    var str = strToBytes(buffer);
    str.replace('hello', '');
    // How to write ???
});

but don't understand how to write it to file:

through2 can help you. The first example code pretty much shows exactly what you want to do. — robertklep
– robertklep, Commented Aug 19, 2017 at 20:14

Michał Karpacki · Accepted Answer · 2018-04-04 15:24:33Z

13

You should use transform stream and use pipes like this:

fs.createReadStream('input/file.txt')
     .pipe(new YourTransformStream())
     .pipe(fs.createWriteStream('output/file.txt'))

Then it's just a matter of implementing the transform stream as in this doc

You can also make this easier for you using scramjet like this:

fs.createReadStream('input/file.txt')
     .pipe(new StringStream('utf-8'))
     .split('\n')                                          // split every line
     .map(async (line) => await makeYourChangesTo(line))   // update the lines
     .join('\n')                                           // join again
     .pipe(fs.createWriteStream('output/file.txt'))

Which I suppose is easier than doing that manually.

answered Apr 4, 2018 at 15:24

Michał Karpacki

2,66624 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dreams Over a year ago

how to do this synchronously, as in how do i know when this has completed and then return back to the calling function?

Michał Karpacki Over a year ago

Well, you don't need to do it synchronously and with streams it is not the point. You can use async/await and then just return a promise that resolves on onFinished. If you want, create a new question regarding this and post it here - I'll answer in more detail. :)

Collectives™ on Stack Overflow

node.js modify file data stream?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related