Concatinate 2 files both from stdin

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Concatinate 2 files both from stdin

Josh Steiner
Hello,

I'm pretty new to SoX and I've been digging through docs and example tutorials with no luck.  

An app I'm working on needs to concatenate some audio files without ever writing them to disk*.  I need to be able to pipe the data to the sox command and have it read in both files from stdin.  

As a test, here is what I have tried:

cat take1.aiff take2.aiff | sox -t aiff - output.aiff

the output.aiff contains only the contents of take1.aiff ?

Any tips or pointers would be greatly appreciated.

-Josh

*long story short, The app (StoryMaker, https://github.com/guardianproject/mrapp) stores it's media in a kind of encrypted filesystem (IOCipher) that doesn't actually get mounted to the system as this is running on an Android device.  There is a command to cat the file contents out from the encrypted fs to stdout, it's important that we don't write the contents to the disk unencrypted for security reasons.

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Concatinate 2 files both from stdin

Ulrich Klauer-2
Josh Steiner wrote:

> An app I'm working on needs to concatenate some audio files without ever
> writing them to disk*.  I need to be able to pipe the data to the sox
> command and have it read in both files from stdin.

> As a test, here is what I have tried:
> cat take1.aiff take2.aiff | sox -t aiff - output.aiff
> the output.aiff contains only the contents of take1.aiff ?

Either use headerless audio (-t raw), or the special pipe input filenames:
   sox "| cat take1.aiff" "| cat take2.aiff" output.aiff

Ulrich


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Concatinate 2 files both from stdin

Jan Stary
In reply to this post by Josh Steiner
On Jul 18 18:53:33, [hidden email] wrote:
> An app I'm working on needs to concatenate some audio files without ever
> writing them to disk*.  I need to be able to pipe the data to the sox
> command and have it read in both files from stdin.

Why do you need SoX to read the input from stdin?
What can't you have SoX simply read them from disk?

> As a test, here is what I have tried:
> cat take1.aiff take2.aiff | sox -t aiff - output.aiff

Here, `cat' is reading the input from disk.
Why can't SoX also read then from disk? Then,

        sox take1.aiff take2.aiff output.aiff

as the example in the 'Introduction' section
of the manpage would tell you.

Also, you are writing outout.aiff to disk
- isn't that precisely what you want to avoid?

> the output.aiff contains only the contents of take1.aiff ?

I think that the 'sox -t aiff', reading stdin,
recognizes the beginning and end of take1.aiff
and just stops there, knowing the length of data
(from the aiff header) has been processed.

This would change with --ignore-length (see the manpage),
but then you would have the non-audio data
of the second aiff header in your output.

> The app (StoryMaker,
> https://github.com/guardianproject/mrapp) stores it's media in a kind of
> encrypted filesystem (IOCipher) that doesn't actually get mounted to the
> system as this is running on an Android device.

So you are running SoX on Android? Is is this port?
https://github.com/Kyborg2011/SoxPlayer

Also, do Android apps have some inherited incapability
of simply writing to a mountd filesystem?

> There is a command to cat the file contents out
> from the encrypted fs to stdout,

Ah, so the `cat' above is the only way to read files
from that "filesystem"?

> it's important that we don't write the contents
> to the disk unencrypted for security reasons.
              ^^^^^^^^^^^

The IOCipher surely provides a way to _write_
(encrypted) data, to the filesystem, right?


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Concatinate 2 files both from stdin

Josh Steiner



On Fri, Jul 19, 2013 at 2:38 AM, Jan Stary <[hidden email]> wrote:
On Jul 18 18:53:33, [hidden email] wrote:
> An app I'm working on needs to concatenate some audio files without ever
> writing them to disk*.  I need to be able to pipe the data to the sox
> command and have it read in both files from stdin.

Why do you need SoX to read the input from stdin?
What can't you have SoX simply read them from disk?

> As a test, here is what I have tried:
> cat take1.aiff take2.aiff | sox -t aiff - output.aiff

Here, `cat' is reading the input from disk.
Why can't SoX also read then from disk? Then,

        sox take1.aiff take2.aiff output.aiff

as the example in the 'Introduction' section
of the manpage would tell you.

I think my example muddied the discussion.  The key is that at no point does our media exist on disk as a file.  We have a utility which will cat from within the encrypted fs to stdout, i need to concat 2 files together using that.  Ignore the command line I offered as an example :)
   

Also, you are writing outout.aiff to disk
- isn't that precisely what you want to avoid?

> the output.aiff contains only the contents of take1.aiff ?

I think that the 'sox -t aiff', reading stdin,
recognizes the beginning and end of take1.aiff
and just stops there, knowing the length of data
(from the aiff header) has been processed.

That makes perfect sense, thanks!
 

This would change with --ignore-length (see the manpage),
but then you would have the non-audio data
of the second aiff header in your output.

> The app (StoryMaker,
> https://github.com/guardianproject/mrapp) stores it's media in a kind of
> encrypted filesystem (IOCipher) that doesn't actually get mounted to the
> system as this is running on an Android device.

So you are running SoX on Android? Is is this port?
https://github.com/Kyborg2011/SoxPlayer

I believe it's this:


which is included in from:

 
Also, do Android apps have some inherited incapability
of simply writing to a mountd filesystem?

No, you have full access to the regular file system, but IOCipher cannot be mounted as a real filesystem unless the user has root.    The library provides a nice java.io.File interface if you are in java, but in this case we need to access files stored within from a shelled out sox (and ffmpeg) process.  This is where the IOCipher library is incomplete.
 

> There is a command to cat the file contents out
> from the encrypted fs to stdout,

Ah, so the `cat' above is the only way to read files
from that "filesystem"?

there is a custom command called "sqlfscat" which let's you cat our the contents of a file within the fs.  And yes, unless you can mount it as a FUSE fs or access it through the Java lib, you can't get access to the fs contents.
 

> it's important that we don't write the contents
> to the disk unencrypted for security reasons.
              ^^^^^^^^^^^

The IOCipher surely provides a way to _write_
(encrypted) data, to the filesystem, right?

Only through FUSE or the Java API for now.  But this is pre-beta so a lot can change :)

My current plan is to setup two named pipes that I feed data from the IOCipher fs and have sox read from those.   Something like what's described here:


This will work in 95% of cases, the only issue is some power users will root and force the app's data dirs to be on the SD card, which is Fat32 and hence won't support a mkfifo pipe.  Good enough for beta, but I need to find a better solution for the public release.  

Perhaps long term using something like JUDS (https://github.com/mcfunley/juds) and using domain sockets...

Thanks,

Josh
 


On Fri, Jul 19, 2013 at 2:38 AM, Jan Stary <[hidden email]> wrote:
On Jul 18 18:53:33, [hidden email] wrote:
> An app I'm working on needs to concatenate some audio files without ever
> writing them to disk*.  I need to be able to pipe the data to the sox
> command and have it read in both files from stdin.

Why do you need SoX to read the input from stdin?
What can't you have SoX simply read them from disk?

> As a test, here is what I have tried:
> cat take1.aiff take2.aiff | sox -t aiff - output.aiff

Here, `cat' is reading the input from disk.
Why can't SoX also read then from disk? Then,

        sox take1.aiff take2.aiff output.aiff

as the example in the 'Introduction' section
of the manpage would tell you.

Also, you are writing outout.aiff to disk
- isn't that precisely what you want to avoid?

> the output.aiff contains only the contents of take1.aiff ?

I think that the 'sox -t aiff', reading stdin,
recognizes the beginning and end of take1.aiff
and just stops there, knowing the length of data
(from the aiff header) has been processed.

This would change with --ignore-length (see the manpage),
but then you would have the non-audio data
of the second aiff header in your output.

> The app (StoryMaker,
> https://github.com/guardianproject/mrapp) stores it's media in a kind of
> encrypted filesystem (IOCipher) that doesn't actually get mounted to the
> system as this is running on an Android device.

So you are running SoX on Android? Is is this port?
https://github.com/Kyborg2011/SoxPlayer

Also, do Android apps have some inherited incapability
of simply writing to a mountd filesystem?

> There is a command to cat the file contents out
> from the encrypted fs to stdout,

Ah, so the `cat' above is the only way to read files
from that "filesystem"?

> it's important that we don't write the contents
> to the disk unencrypted for security reasons.
              ^^^^^^^^^^^

The IOCipher surely provides a way to _write_
(encrypted) data, to the filesystem, right?


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Concatinate 2 files both from stdin

Josh Steiner
In reply to this post by Ulrich Klauer-2
Of course, I should have thought about the fact that AIFF isn't headerless, thanks.

-Josh


On Thu, Jul 18, 2013 at 8:02 PM, Ulrich Klauer <[hidden email]> wrote:
Josh Steiner wrote:

> An app I'm working on needs to concatenate some audio files without ever
> writing them to disk*.  I need to be able to pipe the data to the sox
> command and have it read in both files from stdin.

> As a test, here is what I have tried:
> cat take1.aiff take2.aiff | sox -t aiff - output.aiff
> the output.aiff contains only the contents of take1.aiff ?

Either use headerless audio (-t raw), or the special pipe input filenames:
   sox "| cat take1.aiff" "| cat take2.aiff" output.aiff

Ulrich


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Concatinate 2 files both from stdin

Jan Stary
In reply to this post by Josh Steiner
On Jul 19 09:48:12, [hidden email] wrote:

> > Here, `cat' is reading the input from disk.
> > Why can't SoX also read then from disk? Then,
> >
> >         sox take1.aiff take2.aiff output.aiff
> >
> > as the example in the 'Introduction' section
> > of the manpage would tell you.
> >
> I think my example muddied the discussion.  The key is that at no point
> does our media exist on disk as a file.  We have a utility which will cat
> from within the encrypted fs to stdout, i need to concat 2 files together
> using that.  Ignore the command line I offered as an example :)

OK; just to be completely sure though:
this undisclosed utility isthe only way
to read from the "filesystem".

> > Also, you are writing outout.aiff to disk
> > - isn't that precisely what you want to avoid?

So what are you going to do with the output?

> > So you are running SoX on Android? Is is this port?
> > https://github.com/Kyborg2011/SoxPlayer
>
>
> I believe it's this:

(wait, you don't know?)

> git://sox.git.sourceforge.net/gitroot/sox/sox
> which is included in from:
> https://github.com/guardianproject/android-ffmpeg

Have you used in other contexts too, beside
this application? How well is it working?

> > Also, do Android apps have some inherited incapability
> > of simply writing to a mountd filesystem?
>
> No, you have full access to the regular file system, but IOCipher cannot be
> mounted as a real filesystem unless the user has root.

Why is it that you are using this cumbersome "filesystem"?
Isn't any of the encrypting librarires ported to Android,
which you could just use in your application to encrypt
the files, and then write them encrypted to the normal filesystem?

> The library
> provides a nice java.io.File interface if you are in java, but in this case
> we need to access files stored within from a shelled out sox (and ffmpeg)
> process.  This is where the IOCipher library is incomplete.

Ugh.

> >
> > > There is a command to cat the file contents out
> > > from the encrypted fs to stdout,
> >
> > Ah, so the `cat' above is the only way to read files
> > from that "filesystem"?
> >
>
> there is a custom command called "sqlfscat"
> which let's you cat our the contents of a file within the fs.

"custom" meaning you had to write it?

> And yes, unless you can mount it as a FUSE fs
> or access it through the Java lib, you can't get
> access to the fs contents.

Ech, I fail to see why anyone would use that,
but it's your application.

> >
> > > it's important that we don't write the contents
> > > to the disk unencrypted for security reasons.
> >               ^^^^^^^^^^^
> > The IOCipher surely provides a way to _write_
> > (encrypted) data, to the filesystem, right?
> >
> Only through FUSE or the Java API for now.
> But this is pre-beta so a lot can change :)

So you are using this "filesystem" that is only
accessible through a Java API, but do not write
your app in Java?

> My current plan is to setup two named pipes

named pipes means creating them in the filesystem,
therefore writing to the filesystem,
which you cannot do, right?

> that I feed data from the IOCipher fs
> and have sox read from those.

If SoX can read from a named pipe in the filesystem,
why cannot it simply read the input file too?


        Jan


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Concatinate 2 files both from stdin

Josh Steiner

On Fri, Jul 19, 2013 at 10:16 AM, Jan Stary <[hidden email]> wrote:
On Jul 19 09:48:12, [hidden email] wrote:
> > Here, `cat' is reading the input from disk.
> > Why can't SoX also read then from disk? Then,
> >
> >         sox take1.aiff take2.aiff output.aiff
> >
> > as the example in the 'Introduction' section
> > of the manpage would tell you.
> >
> I think my example muddied the discussion.  The key is that at no point
> does our media exist on disk as a file.  We have a utility which will cat
> from within the encrypted fs to stdout, i need to concat 2 files together
> using that.  Ignore the command line I offered as an example :)

OK; just to be completely sure though:
this undisclosed utility isthe only way
to read from the "filesystem".

It's called sqlfscat and it was written yesterday :)  you can either use that, mount it using FUSE or use the Java apis.
 

> > Also, you are writing outout.aiff to disk
> > - isn't that precisely what you want to avoid?

So what are you going to do with the output?

We'd have to develop a utility like sqlfscat that writes a file into the fs
 
> > So you are running SoX on Android? Is is this port?
> > https://github.com/Kyborg2011/SoxPlayer
>
>
> I believe it's this:

(wait, you don't know?)

That is in fact the submodule we use :)
 
Have you used in other contexts too, beside
this application? How well is it working?

No, this is my first time using SoX on Android.  So far we just use it to do some audio cross fading and concatenation, so nothing complex.  So far so good.  Down the line we will definitely be exploring some more sophisticated audio editing.

 

> > Also, do Android apps have some inherited incapability
> > of simply writing to a mountd filesystem?
>
> No, you have full access to the regular file system, but IOCipher cannot be
> mounted as a real filesystem unless the user has root.

Why is it that you are using this cumbersome "filesystem"?

It's cumbersome because it's in active development right now.   I'm using it because I'm part of the group that is developing it (The Guardian Project)... StoryMaker is being used as a test bed for this new library.  The promise of IOCipher is that for the 99% of Android apps that need a app level secure filesystem IOCipher is a drop in replacement, as long as you don't do crazy stuff like shell out to ported linux command line utils like we do in StoryMaker... so we are a bit of an outlier.
 
 
Isn't any of the encrypting librarires ported to Android,
which you could just use in your application to encrypt
the files, and then write them encrypted to the normal filesystem?

Sure, we could use GPG to encrypt each file, the goal of IOCipher is to be more transparent to the dev though, and if you stay in Java space it succeeds quite nicely.  Clearly I'm running into the areas where it needs some more work.
 

> The library
> provides a nice java.io.File interface if you are in java, but in this case
> we need to access files stored within from a shelled out sox (and ffmpeg)
> process.  This is where the IOCipher library is incomplete.

Ugh.

> >
> > > There is a command to cat the file contents out
> > > from the encrypted fs to stdout,
> >
> > Ah, so the `cat' above is the only way to read files
> > from that "filesystem"?
> >
>
> there is a custom command called "sqlfscat"
> which let's you cat our the contents of a file within the fs.

"custom" meaning you had to write it?

Not me, but the IOCipher guys did.
 

> And yes, unless you can mount it as a FUSE fs
> or access it through the Java lib, you can't get
> access to the fs contents.

Ech, I fail to see why anyone would use that,
but it's your application.

If you are a Java developer it's dead simple.  As in, it takes about 3 minutes to integrate and you don't have to change any logic in your app about how it deals with the fs.

 

> >
> > > it's important that we don't write the contents
> > > to the disk unencrypted for security reasons.
> >               ^^^^^^^^^^^
> > The IOCipher surely provides a way to _write_
> > (encrypted) data, to the filesystem, right?
> >
> Only through FUSE or the Java API for now.
> But this is pre-beta so a lot can change :)

So you are using this "filesystem" that is only
accessible through a Java API, but do not write
your app in Java?

> My current plan is to setup two named pipes

named pipes means creating them in the filesystem,
therefore writing to the filesystem,
which you cannot do, right?

But the data does not get written to disk.  This is what matters.  We are developing this to be hardened against later forensic analyses for people doing journalism work in environments where you need to be concerned about what you record.  When we are done with this integration, you uninstall the app and and there are no traces of the media you created that can be recovered. 
 

> that I feed data from the IOCipher fs
> and have sox read from those.

If SoX can read from a named pipe in the filesystem,
why cannot it simply read the input file too?

The app will open a named pipe, connect sox to read from it and feeds data to the pipe from Java space using the IOCipher Java apis to read the data.    

This is drifting pretty far of topic I think.

Thanks for all the feedback, Jan.

-Josh
 


        Jan


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Concatinate 2 files both from stdin

Ulrich Klauer-2
Josh Steiner wrote:

> git://sox.git.sourceforge.net/gitroot/sox/sox

You should update this to git://git.code.sf.net/p/sox/code -  
SourceForge has recently upgraded its platform, which included setting  
up new Git repositories, and the old repos are still available for an  
indeterminate time, but no longer updated.

Ulrich


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users