Compressie

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Compressie

hendrikus godvliet
For a couple of years i am working with sox. One of the thinks i don't get along with is the compand ( compression ).
I tried many configureration but not one of them satisfied me.

What i am looking for is a verry lightfull compresie for a singer ( female and male vocale's)

Please if aneyone can help me i would appreciate


Hendrikus



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compressie

Eric Wong
Hendrikus Godvliet <[hidden email]> wrote:
> For a couple of years i am working with sox. One of the thinks i don't get
> along with is the compand ( compression ).
> I tried many configureration but not one of them satisfied me.
>
> What i am looking for is a verry lightfull compresie for a singer ( female
> and male vocale's)

Can you tell us what settings you've tried?
Can you share an audio sample you're working with?
Can you describe what effect(s) you're trying to achieve?

The compander requires tuning for every recording and every goal.

I've only been using editing audio (and using SoX) for a few months, but
I've found the following posts on the ML very helpful to understanding
the compander:

  http://mid.gmane.org/20090829011522.cf27e6ee.fmiser@...
  http://mid.gmane.org/20091231022430.aaa510b3.fmiser@...

For my own uses (editing audience recordings of rock/metal concerts),
I'll (intentionally) setup my compander(s) to brickwall audience noise.
So my compander settings probably won't work for you :)

A few notes of my own:

* the "stats" effect is important for getting the levels right

* Combine trim with stats to get the levels for sections you're
  interested in

* a 0,0.02 attack,decay to bring down peaks to the peak level
  of music works well for cutting isolated hand clapping.

* a $SUBSECOND_MAYBE_ZERO,$SEVERAL_SECONDS_VERY_LONG attack,decay
  works well for cheering in between tracks.  If you only limit
  the volume to the levels at the _end_ of the companded track,
  you can probably use an infinitely long decay time.
  I sometimes use a soft-knee here, too.

* Sometimes, I'll run 2 companders in sequence:
  One with a quick attack,decay and higher levels (for clapping)
  One with a slow attack,decay and lower levels (for cheering/screaming)

  I /think/ the quicker, less-aggressive compander should
  always run first.  2 companders lets me worry less about
  getting the attack for the second compander.

  I haven't used more than 2 compander at once on the same channel,
  but it's possible and I may do so one day :)

* Remember to set initial-volume-dB.  It's important if you're
  (like me) using trim to split a track into 3+ parts, compand
  the middle section(s), and rejoin the pieces back together.
  Leaving it unset (at 0 dB FS) can make the compander start
  working even when you don't want it to.

* I haven't figured out how to use the delay parameter
  effectively.

  I think something like the TAP Scaling Limiter (tap-plugins
  LADSPA package) might do better (but TAP-limiter introduces some
  latency)


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compressie

hendrikus godvliet
Eric

Thank you for
your very comprehensive informationI'm studying it now and it looks as though I have many bettere results with my compand institutions.

Together with your info i found this also helpfull:

http://www.wilfridwong.com/articles/audio-compression-for-beginners-setting-compressor-threshold-ratio-knee-attack-release-and-more/

http://www.wilfridwong.com/articles/audio-compression-for-beginners-setting-compressor-threshold-ratio-knee-attack-release-and-more/applying-different-vintage-compressor-settings-part-2/


Hendrikus



On Sat, Jul 7, 2012 at 9:42 AM, Eric Wong <[hidden email]> wrote:
Hendrikus Godvliet <[hidden email]> wrote:
> For a couple of years i am working with sox. One of the thinks i don't get
> along with is the compand ( compression ).
> I tried many configureration but not one of them satisfied me.
>
> What i am looking for is a verry lightfull compresie for a singer ( female
> and male vocale's)

Can you tell us what settings you've tried?
Can you share an audio sample you're working with?
Can you describe what effect(s) you're trying to achieve?

The compander requires tuning for every recording and every goal.

I've only been using editing audio (and using SoX) for a few months, but
I've found the following posts on the ML very helpful to understanding
the compander:

  http://mid.gmane.org/20090829011522.cf27e6ee.fmiser@...
  http://mid.gmane.org/20091231022430.aaa510b3.fmiser@...

For my own uses (editing audience recordings of rock/metal concerts),
I'll (intentionally) setup my compander(s) to brickwall audience noise.
So my compander settings probably won't work for you :)

A few notes of my own:

* the "stats" effect is important for getting the levels right

* Combine trim with stats to get the levels for sections you're
  interested in

* a 0,0.02 attack,decay to bring down peaks to the peak level
  of music works well for cutting isolated hand clapping.

* a $SUBSECOND_MAYBE_ZERO,$SEVERAL_SECONDS_VERY_LONG attack,decay
  works well for cheering in between tracks.  If you only limit
  the volume to the levels at the _end_ of the companded track,
  you can probably use an infinitely long decay time.
  I sometimes use a soft-knee here, too.

* Sometimes, I'll run 2 companders in sequence:
  One with a quick attack,decay and higher levels (for clapping)
  One with a slow attack,decay and lower levels (for cheering/screaming)

  I /think/ the quicker, less-aggressive compander should
  always run first.  2 companders lets me worry less about
  getting the attack for the second compander.

  I haven't used more than 2 compander at once on the same channel,
  but it's possible and I may do so one day :)

* Remember to set initial-volume-dB.  It's important if you're
  (like me) using trim to split a track into 3+ parts, compand
  the middle section(s), and rejoin the pieces back together.
  Leaving it unset (at 0 dB FS) can make the compander start
  working even when you don't want it to.

* I haven't figured out how to use the delay parameter
  effectively.

  I think something like the TAP Scaling Limiter (tap-plugins
  LADSPA package) might do better (but TAP-limiter introduces some
  latency)


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compressie

hendrikus godvliet
In reply to this post by Eric Wong
Eric

<* the "stats" effect is important for getting the levels right

<* Combine trim with stats to get the levels for sections you're interested in

Could you give me a more detailed info who you work around with this - this is most intresting to me?

Hendrikus


On Sat, Jul 7, 2012 at 9:42 AM, Eric Wong <[hidden email]> wrote:
Hendrikus Godvliet <[hidden email]> wrote:
> For a couple of years i am working with sox. One of the thinks i don't get
> along with is the compand ( compression ).
> I tried many configureration but not one of them satisfied me.
>
> What i am looking for is a verry lightfull compresie for a singer ( female
> and male vocale's)

Can you tell us what settings you've tried?
Can you share an audio sample you're working with?
Can you describe what effect(s) you're trying to achieve?

The compander requires tuning for every recording and every goal.

I've only been using editing audio (and using SoX) for a few months, but
I've found the following posts on the ML very helpful to understanding
the compander:

  http://mid.gmane.org/20090829011522.cf27e6ee.fmiser@...
  http://mid.gmane.org/20091231022430.aaa510b3.fmiser@...

For my own uses (editing audience recordings of rock/metal concerts),
I'll (intentionally) setup my compander(s) to brickwall audience noise.
So my compander settings probably won't work for you :)

A few notes of my own:

* the "stats" effect is important for getting the levels right

* Combine trim with stats to get the levels for sections you're
  interested in

* a 0,0.02 attack,decay to bring down peaks to the peak level
  of music works well for cutting isolated hand clapping.

* a $SUBSECOND_MAYBE_ZERO,$SEVERAL_SECONDS_VERY_LONG attack,decay
  works well for cheering in between tracks.  If you only limit
  the volume to the levels at the _end_ of the companded track,
  you can probably use an infinitely long decay time.
  I sometimes use a soft-knee here, too.

* Sometimes, I'll run 2 companders in sequence:
  One with a quick attack,decay and higher levels (for clapping)
  One with a slow attack,decay and lower levels (for cheering/screaming)

  I /think/ the quicker, less-aggressive compander should
  always run first.  2 companders lets me worry less about
  getting the attack for the second compander.

  I haven't used more than 2 compander at once on the same channel,
  but it's possible and I may do so one day :)

* Remember to set initial-volume-dB.  It's important if you're
  (like me) using trim to split a track into 3+ parts, compand
  the middle section(s), and rejoin the pieces back together.
  Leaving it unset (at 0 dB FS) can make the compander start
  working even when you don't want it to.

* I haven't figured out how to use the delay parameter
  effectively.

  I think something like the TAP Scaling Limiter (tap-plugins
  LADSPA package) might do better (but TAP-limiter introduces some
  latency)


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compressie

Eric Wong
Hendrikus Godvliet <[hidden email]> wrote:
> Eric
>
> <* the "stats" effect is important for getting the levels right
>
> <* Combine trim with stats to get the levels for sections you're interested
> in
>
> Could you give me a more detailed info who you work around with this - this
> is most intresting to me?

The "stats" effect will give you the standard peak level (Pk lev dB),
as well as the RMS level, peak, and trough values.

The relationship between the levels for any recording should be as
follows:

        Pk lev dB > RMS Pk dB > RMS lev DB > RMS Tr dB

If you're doing top-down compression (to quiet down loud parts), you can
start by setting compressor to something lower than Pk lev dB.  If the
peaks are still too loud for you, then you can try getting it near
RMS Pk dB, and finally maybe even closer to RMS lev DB.

You can also try to add bottom-up compression (making quiet parts
louder).  For that, you want to raise the lower levels (above the
RMS trough) higher, but be careful not to bring the noise floor
up too high, either.

With trim, you can isolate the "stats" effect to only a certain area.

So if you're happy with the first 5s and last 5s of a 15s recording, but
unhappy with the levels in the middle 5s, you could so something like
this:

    sox $INPUT -n trim 0 =5 stats
    sox $INPUT -n trim 5 =10 stats # the part you're unhappy with
    sox $INPUT -n trim 10 =15 stats

Since we're assuming you like the surrounding parts, you'll want to
build a compander mapping that brings the middle (5 =10) part down (or
up) to the levels of the parts surrounding it.

You can either run the compander on the entire 15s recording, or split
out the sections (via trim), compand the middle, and rejoin them using
sox.

Tuning a compander unfortunately requires a lot of manual settings
that vary between recordings (and even different parts of the same
song).

(I'm new to audio editing myself, so if somebody else on
 this list can correct/improve on what I write, I'd much appreciate it).

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compressie

hendrikus godvliet
Eric

You are very helpful to me with your instructions thank you for it
I'm going to use and lets you know how it goes




Hendrikus




On Thu, Jul 12, 2012 at 10:04 AM, Eric Wong <[hidden email]> wrote:
Hendrikus Godvliet <[hidden email]> wrote:
> Eric
>
> <* the "stats" effect is important for getting the levels right
>
> <* Combine trim with stats to get the levels for sections you're interested
> in
>
> Could you give me a more detailed info who you work around with this - this
> is most intresting to me?

The "stats" effect will give you the standard peak level (Pk lev dB),
as well as the RMS level, peak, and trough values.

The relationship between the levels for any recording should be as
follows:

        Pk lev dB > RMS Pk dB > RMS lev DB > RMS Tr dB

If you're doing top-down compression (to quiet down loud parts), you can
start by setting compressor to something lower than Pk lev dB.  If the
peaks are still too loud for you, then you can try getting it near
RMS Pk dB, and finally maybe even closer to RMS lev DB.

You can also try to add bottom-up compression (making quiet parts
louder).  For that, you want to raise the lower levels (above the
RMS trough) higher, but be careful not to bring the noise floor
up too high, either.

With trim, you can isolate the "stats" effect to only a certain area.

So if you're happy with the first 5s and last 5s of a 15s recording, but
unhappy with the levels in the middle 5s, you could so something like
this:

    sox $INPUT -n trim 0 =5 stats
    sox $INPUT -n trim 5 =10 stats # the part you're unhappy with
    sox $INPUT -n trim 10 =15 stats

Since we're assuming you like the surrounding parts, you'll want to
build a compander mapping that brings the middle (5 =10) part down (or
up) to the levels of the parts surrounding it.

You can either run the compander on the entire 15s recording, or split
out the sections (via trim), compand the middle, and rejoin them using
sox.

Tuning a compander unfortunately requires a lot of manual settings
that vary between recordings (and even different parts of the same
song).

(I'm new to audio editing myself, so if somebody else on
 this list can correct/improve on what I write, I'd much appreciate it).

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Mew, Peter

Hi

Is there a compare function in Sox, to compare two audio files to see if they are the same, byte for byte

 

Thanks

-pm

 


---------------------------------------------------------------------




Music from EMI

This e-mail including any attachments is confidential and may be legally privileged. If you have received it in error please advise the sender immediately by return email and then delete it from your system. The unauthorised use, distribution, copying or alteration of this email is strictly forbidden. If you need assistance please contact us on +44 20 7795 7000.

This email is from a unit or subsidiary of EMI Group Limited.

Registered Office: 27 Wrights Lane, London W8 5SW

Registered in England No 229231.


---------------------------------------------------------------------

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Jan Stary
On Jul 12 11:43:33, Mew, Peter wrote:
> Hi
>
> Is there a compare function in Sox, to compare two audio files to see if
> they are the same, byte for byte

You don't need sox (or any audio software) for that
- just use diff(1)


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Rob Sykes



From: Jan Stary <[hidden email]>

On Jul 12 11:43:33, Mew, Peter wrote:
> Hi
>
> Is there a compare function in Sox, to compare two audio files to see if
> they are the same, byte for byte

You don't need sox (or any audio software) for that
- just use diff(1)

True, but sometimes there are header differences you may want to ignore.

I often use something like:

sox -m -v -1 file1 file2 -n stat (or stats)

to check for no (or very small) differences (that is, if I'm already sure they're the same length; if not, soxi or sox with -V can be used to check this).

Cheers,
Rob

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Rafal Maszkowski
In reply to this post by Jan Stary
On Thu, Jul 12, 2012 at 01:49:14PM +0200, Jan Stary wrote:
> On Jul 12 11:43:33, Mew, Peter wrote:
> > Is there a compare function in Sox, to compare two audio files to see if
> > they are the same, byte for byte
> You don't need sox (or any audio software) for that
> - just use diff(1)

Or cmp.
(For more complicated cases MPEG-7 would be nice to have...)

R.
--
Жить надо так, чтобы тебя помнили и сволочи. (Фаина Георгиевна Раневская)
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Mew, Peter
In reply to this post by Rob Sykes

Exactly.

To compare a RAW file with a WAV file that has the same data, and you want to check the data really is the same

At the moment I convert the WAV file to RAW with sox, make an md5 checksum for each of the RAW files and compare them.

I’d like to cut out the md5 process and do the comparison all within sox.

And I’d like it to be cross platform

 

Cheers

-pm

 

From: robs [mailto:[hidden email]]
Sent: Thu 12 Jul 2012 13:06
To: [hidden email]
Subject: Re: [SoX-users] Compare

 

 

 


From: Jan Stary <[hidden email]>

 

On Jul 12 11:43:33, Mew, Peter wrote:
> Hi
>
> Is there a compare function in Sox, to compare two audio files to see if
> they are the same, byte for byte

You don't need sox (or any audio software) for that
- just use diff(1)

 

True, but sometimes there are header differences you may want to ignore.

 

I often use something like:

 

sox -m -v -1 file1 file2 -n stat (or stats)

 

to check for no (or very small) differences (that is, if I'm already sure they're the same length; if not, soxi or sox with -V can be used to check this).

 

Cheers,

Rob


---------------------------------------------------------------------




Music from EMI

This e-mail including any attachments is confidential and may be legally privileged. If you have received it in error please advise the sender immediately by return email and then delete it from your system. The unauthorised use, distribution, copying or alteration of this email is strictly forbidden. If you need assistance please contact us on +44 20 7795 7000.

This email is from a unit or subsidiary of EMI Group Limited.

Registered Office: 27 Wrights Lane, London W8 5SW

Registered in England No 229231.


---------------------------------------------------------------------

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Jan Stary
On Jul 12 14:12:41, Mew, Peter wrote:
> To compare a RAW file with a WAV file that has the same data, and you
> want to check the data really is the same

Well, that's not the same as checking whether
"the files are the same byte by byte" as you originaly stated it.
But comparing just the audio bytes makes more sense here of course.

> At the moment I convert the WAV file to RAW with sox, make an md5
> checksum for each of the RAW files and compare them.

Why do you compare the md5's and not the raw data itself?

> I'd like to cut out the md5 process and do the comparison all within
> sox. And I'd like it to be cross platform

Convert the WAV file to the raw format used in the RAW file
and cmp(1) or diff(1) the two copies of the raw audio data.

        Jan


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Dave Graff
In reply to this post by Mew, Peter

On Jul 12, 2012, at 9:12 AM, Mew, Peter wrote:

> Exactly.
> To compare a RAW file with a WAV file that has the same data, and you want to check the data really is the same
> At the moment I convert the WAV file to RAW with sox, make an md5 checksum for each of the RAW files and compare them.
> I’d like to cut out the md5 process and do the comparison all within sox.

Using md5 on the sample data is actually likely to be as efficient as anything you could do "all within sox".  It's just a matter of streamlining a process that extracts md5 checksums on just the sample data for any given pair (or group) of files.

(If you happen to face a situation where a pool of, say, 50 files might contain an indeterminate set of duplicates, triplicates, etc, then md5 checksums are the only effective way to work that out -- pair-wise comparisons would be a real waste of time.)

> And I’d like it to be cross platform

Do you mean you'd like it "do the right thing" on raw files regardless of their "native" byte order?  (As I understand it, mswav/RIFF format dictates little-endian, regardless of platform.)

FWIW, flac compression includes a step of computing the sample md5 checksum for the uncompressed data and storing that in the flac header; I don't see anything in sox that provides access to that (it might be nice for soxi to include that in its output for flac files).

If you store your data as flac, there are library modules available for the popular scripting languages (perl for sure, presumably ruby and python as well) that make it easy to fetch the sample md5 value from the flac header.

If you don't go that route, I think the best thing would be a general-purpose script that uses sox to extract md5 checksums from any given list of files.  The script would have to provide a means (typically a command-line option) for telling sox to invert the byte order of raw data when appropriate.

        Dave Graff


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Mew, Peter
Hi
By Cross platform, I mean both MAC and PC, which is why a sox solution would be good
does sox "extract md5 checksums"?

thanks
-pm


-----Original Message-----
From: Dave Graff [mailto:[hidden email]]
Sent: Thu 12/07/2012 15:16
To: [hidden email]
Cc: Dave Graff
Subject: Re: [SoX-users] Compare
 

On Jul 12, 2012, at 9:12 AM, Mew, Peter wrote:

> Exactly.
> To compare a RAW file with a WAV file that has the same data, and you want to check the data really is the same
> At the moment I convert the WAV file to RAW with sox, make an md5 checksum for each of the RAW files and compare them.
> I'd like to cut out the md5 process and do the comparison all within sox.

Using md5 on the sample data is actually likely to be as efficient as anything you could do "all within sox".  It's just a matter of streamlining a process that extracts md5 checksums on just the sample data for any given pair (or group) of files.

(If you happen to face a situation where a pool of, say, 50 files might contain an indeterminate set of duplicates, triplicates, etc, then md5 checksums are the only effective way to work that out -- pair-wise comparisons would be a real waste of time.)

> And I'd like it to be cross platform

Do you mean you'd like it "do the right thing" on raw files regardless of their "native" byte order?  (As I understand it, mswav/RIFF format dictates little-endian, regardless of platform.)

FWIW, flac compression includes a step of computing the sample md5 checksum for the uncompressed data and storing that in the flac header; I don't see anything in sox that provides access to that (it might be nice for soxi to include that in its output for flac files).

If you store your data as flac, there are library modules available for the popular scripting languages (perl for sure, presumably ruby and python as well) that make it easy to fetch the sample md5 value from the flac header.

If you don't go that route, I think the best thing would be a general-purpose script that uses sox to extract md5 checksums from any given list of files.  The script would have to provide a means (typically a command-line option) for telling sox to invert the byte order of raw data when appropriate.

        Dave Graff


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users


 ---------------------------------------------------------------------




Music from EMI

This e-mail including any attachments is confidential and may be legally privileged. If you have received it in error please advise the sender immediately by return email and then delete it from your system. The unauthorised use, distribution, copying or alteration of this email is strictly forbidden. If you need assistance please contact us on +44 20 7795 7000.

This email is from a unit or subsidiary of EMI Group Limited.

Registered Office: 27 Wrights Lane, London W8 5SW

Registered in England No 229231.


 ---------------------------------------------------------------------
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users

winmail.dat (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Chris Bagwell
In reply to this post by Mew, Peter
I'm not totally sure I'd recommend this over md5 but its a fun
experiment.  You could use sox to mix 2 audio files together and if
you first invert one signal then identical audio will cancel each
other out.  Min/max amplitude should become zero for that one case.

This worked with "monkey.wav" that comes with sox anyways:

> sox -m monkey.wav "|sox -v -1 monkey.wav -p" -n stat
Samples read:              1612
Length (seconds):      0.201198
Scaled by:         2147483647.0
Maximum amplitude:     0.000000
Minimum amplitude:     0.000000
Midline amplitude:     0.000000
Mean    norm:          0.000000
Mean    amplitude:     0.000000
RMS     amplitude:     0.000000
Maximum delta:         0.000000
Minimum delta:         0.000000
Mean    delta:         0.000000
RMS     delta:         0.000000
Rough   frequency:  -2147483648

Chris

On Thu, Jul 12, 2012 at 8:12 AM, Mew, Peter <[hidden email]> wrote:

> Exactly.
>
> To compare a RAW file with a WAV file that has the same data, and you want
> to check the data really is the same
>
> At the moment I convert the WAV file to RAW with sox, make an md5 checksum
> for each of the RAW files and compare them.
>
> I’d like to cut out the md5 process and do the comparison all within sox.
>
> And I’d like it to be cross platform
>
>
>
> Cheers
>
> -pm
>
>
>
> From: robs [mailto:[hidden email]]
> Sent: Thu 12 Jul 2012 13:06
> To: [hidden email]
> Subject: Re: [SoX-users] Compare
>
>
>
>
>
>
>
> ________________________________
>
> From: Jan Stary <[hidden email]>
>
>
>
> On Jul 12 11:43:33, Mew, Peter wrote:
>> Hi
>>
>> Is there a compare function in Sox, to compare two audio files to see if
>> they are the same, byte for byte
>
> You don't need sox (or any audio software) for that
> - just use diff(1)
>
>
>
> True, but sometimes there are header differences you may want to ignore.
>
>
>
> I often use something like:
>
>
>
> sox -m -v -1 file1 file2 -n stat (or stats)
>
>
>
> to check for no (or very small) differences (that is, if I'm already sure
> they're the same length; if not, soxi or sox with -V can be used to check
> this).
>
>
>
> Cheers,
>
> Rob
>
>
> ---------------------------------------------------------------------
>
>
>
>
> Music from EMI
>
> This e-mail including any attachments is confidential and may be legally
> privileged. If you have received it in error please advise the sender
> immediately by return email and then delete it from your system. The
> unauthorised use, distribution, copying or alteration of this email is
> strictly forbidden. If you need assistance please contact us on +44 20 7795
> 7000.
>
> This email is from a unit or subsidiary of EMI Group Limited.
>
> Registered Office: 27 Wrights Lane, London W8 5SW
>
> Registered in England No 229231.
>
>
> ---------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Sox-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/sox-users
>

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Jan Stary
In reply to this post by Mew, Peter
On Jul 12 16:23:08, Mew, Peter wrote:
> Hi
> By Cross platform, I mean both MAC and PC, which is why a sox solution would be good
> does sox "extract md5 checksums"?

No


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Jan Stary
In reply to this post by Chris Bagwell
On Jul 12 10:39:45, Chris Bagwell wrote:

> I'm not totally sure I'd recommend this over md5 but its a fun
> experiment.  You could use sox to mix 2 audio files together and if
> you first invert one signal then identical audio will cancel each
> other out.  Min/max amplitude should become zero for that one case.
>
> This worked with "monkey.wav" that comes with sox anyways:
>
> > sox -m monkey.wav "|sox -v -1 monkey.wav -p" -n stat
> Samples read:              1612
> Length (seconds):      0.201198
> Scaled by:         2147483647.0
> Maximum amplitude:     0.000000
> Minimum amplitude:     0.000000
> Midline amplitude:     0.000000
> Mean    norm:          0.000000
> Mean    amplitude:     0.000000
> RMS     amplitude:     0.000000
> Maximum delta:         0.000000
> Minimum delta:         0.000000
> Mean    delta:         0.000000
> RMS     delta:         0.000000
> Rough   frequency:  -2147483648

While I agree this is neat (I believe this is used in analog audio
to listen if your speakers are properly calibrated: send the same signal
into both, reverse the phase of one -> zero result),
I think the processing is unnecessary for the purpose
of byte-by-byte comparison. diff will do the same
without any audio processing.

same argument goes for md5: why do it?


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Colin Sheaff-2
On Thu, Jul 12, 2012 at 11:58 AM, Jan Stary <[hidden email]> wrote:

same argument goes for md5: why do it?

An md5 hash is a good short "unique fingerprint" of the data in the file which is why it's used by git and many other programs to determine if files are the same regardless of filename or other metadata. Once the md5 is generated it is much faster to compare md5 hashes than full files, esp. if you're doing many comparisons.


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Jan Stary
On Jul 12 12:27:36, Colin Sheaff wrote:

> On Thu, Jul 12, 2012 at 11:58 AM, Jan Stary <[hidden email]> wrote:
>
> >
> > same argument goes for md5: why do it?
> >
> > An md5 hash is a good short "unique fingerprint" of the data in the file
> which is why it's used by git and many other programs to determine if files
> are the same regardless of filename or other metadata. Once the md5 is
> generated it is much faster to compare md5 hashes than full files, esp. if
> you're doing many comparisons.

Yes, I know what an md5 is. But here your are comparing just two files,
so it is cheapest to just compare them byte by byte - computing the hash
is actually unnecessary extra work.


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users
Reply | Threaded
Open this post in threaded view
|

Re: Compare

Rob Sykes
In reply to this post by Colin Sheaff-2
So if we're restricted to just SoX (which keeps things cross-platform), we can do, for example:

sox -m -v -1 -V file1.wav file2.aiff -n stat

Which, trimmed a bit, gives:

Input File     : 'file1.wav'
Channels       : 1
Sample Rate    : 48000
Precision      : 16-bit
Duration       : 00:00:07.22 = 346426 samples ~ 541.291 CDDA sectors
Sample Encoding: 16-bit Signed Integer PCM
Endian Type    : little

Input File     : 'file2.aiff'
Channels       : 1
Sample Rate    : 48000
Precision      : 24-bit
Duration       : 00:00:07.22 = 346427 samples ~ 541.292 CDDA sectors
Sample Encoding: 24-bit Signed Integer PCM
Endian Type    : big

Maximum amplitude:     0.000000
Minimum amplitude:     0.000000

Not a simple, yes/no answer: as well as checking that the amplitudes are zero, we have to check that the bit-depth, the number of channels and the durations are the same (here, the bit depths are different and one file is one sample longer), but on particular occasions, such differences may or may not be considered important.
               
Rob

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sox-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/sox-users