Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

kewping
Hello all,

any suggestions on where/how to start approaching Data set/ Dimensional
Reduction / PCA (Principal Component Analysis)  in sc?
there seems to be quite little information about it.

The main purpose is to play a different sound for a dataset of 286 ( 286
of column) into smaller dataset (for example 3 dataset (contain of carrier
frequency, modulator frequency, ...)

I need to reduce the dataset of 286 column into smaller dataset and when
play the sound, it only produce |5 or 3 sound.

many thanks!



--
Sent from: http://new-supercollider-mailing-lists-forums-use-these.2681727.n2.nabble.com/SuperCollider-Users-New-Use-this-f2676391.html

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

user_weego
Hi,
Your request is a bit vague without a small code example, but you may want to look into the .clump and .flop methods of lists/arrays to slice your dataset.

lG,
Hannes

typed this on a "smart" keypad.

> On Dec 21, 2017, at 17:32, [hidden email] wrote:
>
> Hello all,
>
> any suggestions on where/how to start approaching Data set/ Dimensional
> Reduction / PCA (Principal Component Analysis)  in sc?
> there seems to be quite little information about it.
>
> The main purpose is to play a different sound for a dataset of 286 ( 286
> of column) into smaller dataset (for example 3 dataset (contain of carrier
> frequency, modulator frequency, ...)
>
> I need to reduce the dataset of 286 column into smaller dataset and when
> play the sound, it only produce |5 or 3 sound.
>
> many thanks!
>
>
>
> --
> Sent from: http://new-supercollider-mailing-lists-forums-use-these.2681727.n2.nabble.com/SuperCollider-Users-New-Use-this-f2676391.html
>
> _______________________________________________
> sc-users mailing list
>
> info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
> archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
> search: http://www.listarc.bham.ac.uk/lists/sc-users/search/


_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

kewping
Thank you for your reply.

I have a dataset (training set) of 286 (column) * 56 (row), meaning:
- 56 training example.
- Each training example have 286 data of different number

My code:

a = CSVFileReader.read("C:\\Users\\ID977\\Data_Coffee.csv",skipEmptyLines:
true).postcs;
b = a.collect { |row| row.collect { |s| s.asFloat } };
b.postcs;

(
Pbind(
        \instrument, \fmgrain1,
  \carrfreq, 10 * Pseq(b).trace / 4,
                \modfreq, Pkey(\carrfreq) / 5,
            \index, 2.5,
        \attack, 0.001,    // and fixed values as desired
        \decay, 0.2,       // for the other parameters
        \dur, 0.1
).trace.play
)

--> I need to generate a sequences of sound of 256.




--
Sent from: http://new-supercollider-mailing-lists-forums-use-these.2681727.n2.nabble.com/SuperCollider-Users-New-Use-this-f2676391.html

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

crucialfelix

There are matrix math implementations in the MathLib quark.

There are some standard ways to calculate PCA:


involving covariance matrix, eigenvalues and what not.  I did this once (but not in sc)
There is a .cov(ariance) method on Matrix in MathLib

If this makes your head hurt then try porting a full solution from some other language:


Hope that helps or gets you started.

- felix



On Thu, Dec 21, 2017 at 11:07 PM, <[hidden email]> wrote:
Thank you for your reply.

I have a dataset (training set) of 286 (column) * 56 (row), meaning:
- 56 training example.
- Each training example have 286 data of different number

My code:

a = CSVFileReader.read("C:\\Users\\ID977\\Data_Coffee.csv",skipEmptyLines:
true).postcs;
b = a.collect { |row| row.collect { |s| s.asFloat } };
b.postcs;

(
Pbind(
        \instrument, \fmgrain1,
        \carrfreq, 10 * Pseq(b).trace / 4,
                \modfreq, Pkey(\carrfreq) / 5,
            \index, 2.5,
        \attack, 0.001,    // and fixed values as desired
        \decay, 0.2,       // for the other parameters
        \dur, 0.1
).trace.play
)

--> I need to generate a sequences of sound of 256.

Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

Gasp
hi there im already part of the fuorum i think cause im recieving so many e mails, but I dont know where to ask questions or find tutorials.
anyone can help me?
Greetings from Chile

2017-12-21 20:15 GMT-03:00 <[hidden email]>:

There are matrix math implementations in the MathLib quark.

There are some standard ways to calculate PCA:


involving covariance matrix, eigenvalues and what not.  I did this once (but not in sc)
There is a .cov(ariance) method on Matrix in MathLib

If this makes your head hurt then try porting a full solution from some other language:


Hope that helps or gets you started.

- felix



On Thu, Dec 21, 2017 at 11:07 PM, <[hidden email]> wrote:
Thank you for your reply.

I have a dataset (training set) of 286 (column) * 56 (row), meaning:
- 56 training example.
- Each training example have 286 data of different number

My code:

a = CSVFileReader.read("C:\\Users\\ID977\\Data_Coffee.csv",skipEmptyLines:
true).postcs;
b = a.collect { |row| row.collect { |s| s.asFloat } };
b.postcs;

(
Pbind(
        \instrument, \fmgrain1,
        \carrfreq, 10 * Pseq(b).trace / 4,
                \modfreq, Pkey(\carrfreq) / 5,
            \index, 2.5,
        \attack, 0.001,    // and fixed values as desired
        \decay, 0.2,       // for the other parameters
        \dur, 0.1
).trace.play
)

--> I need to generate a sequences of sound of 256.


Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

daniel-mayer

Am 22.12.2017 um 01:41 schrieb [hidden email]:

> hi there im already part of the fuorum i think cause im recieving so many e mails, but I dont know where to ask questions or find tutorials.
> anyone can help me?
> Greetings from Chile

Welcome,

this is the right place to ask questions. Just one note: please don't reply to unrelated mails, also not with a replaced topic. Start a thread with a new email with a meaningful subject line, otherwise things get messed up in the archives.

Besides the tutorials of SC's built-in help system, which you find online also ...

http://doc.sccode.org/Browse.html#Tutorials%3EGetting-Started

http://doc.sccode.org/Browse.html#Tutorials%3EMark_Polishook_tutorial


... you find a list of tutorials here:

http://supercollider.github.io/tutorials/


Greetings

Daniel

-----------------------------
www.daniel-mayer.at
-----------------------------





_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

Gasp
Thanks a lot Daniel
greetings

2017-12-21 22:57 GMT-03:00 <[hidden email]>:

Am 22.12.2017 um 01:41 schrieb [hidden email]:

> hi there im already part of the fuorum i think cause im recieving so many e mails, but I dont know where to ask questions or find tutorials.
> anyone can help me?
> Greetings from Chile

Welcome,

this is the right place to ask questions. Just one note: please don't reply to unrelated mails, also not with a replaced topic. Start a thread with a new email with a meaningful subject line, otherwise things get messed up in the archives.

Besides the tutorials of SC's built-in help system, which you find online also ...

http://doc.sccode.org/Browse.html#Tutorials%3EGetting-Started

http://doc.sccode.org/Browse.html#Tutorials%3EMark_Polishook_tutorial


... you find a list of tutorials here:

http://supercollider.github.io/tutorials/


Greetings

Daniel

-----------------------------
www.daniel-mayer.at
-----------------------------

Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

Nathan Ho
In reply to this post by kewping
On 2017-12-21 08:32, [hidden email] wrote:
> Hello all,
>
> any suggestions on where/how to start approaching Data set/ Dimensional
> Reduction / PCA (Principal Component Analysis)  in sc?
> there seems to be quite little information about it.

hi kewping,

it depends on the complexity of your application and how much you value
efficiency, but i'd suggest doing scientific computation in an
established scientific computation environment (e.g. NumPy/SciPy) rather
than rolling your own. simplest way to get sclang talking to external
languages is using .unixCmd.


nathan

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

kewping
Hi,

Thank you for your reply.

Can you elaborate more on .unixCmd?

As I know "NumPy/SciPy" is used in Python.



--
Sent from: http://new-supercollider-mailing-lists-forums-use-these.2681727.n2.nabble.com/SuperCollider-Users-New-Use-this-f2676391.html

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

Nathan Ho
On 2017-12-22 04:27, [hidden email] wrote:
> Hi,
>
> Thank you for your reply.
>
> Can you elaborate more on .unixCmd?

hi kewping,

sure thing. since part of your application is written in sclang and part
would be written in (as an example) python, you need a way for the two
languages to talk to each other. the easiest way to do this is to run
python as a subprocess of sclang, e.g. "python myscript.py".unixCmd.


nathan

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

grahamsw
I'm fairly new to SC, but I know that it's not the language I want to do any serious data analysis in. My own interest is primarily in sonification, so that involves explicit non-sonic data being transformed into control signals.

My thinking at the moment is that it's best to separate the two cleanly - start with instruments that have clear, sonically relevant control parameters, and translate the data stream, whatever shape it comes in, into control signals. 

I'm using Python externally to process data streams, generating streams of sonic control signals, and sending OSC signals to SC.

Seems to work, but I'm aware that there is a ton of experience out there and I'm very interested in other people's ideas.

One thing external OSC has going for it is that it scales.

On Fri, Dec 22, 2017 at 7:13 PM <[hidden email]> wrote:
On 2017-12-22 04:27, [hidden email] wrote:
> Hi,
>
> Thank you for your reply.
>
> Can you elaborate more on .unixCmd?

hi kewping,

sure thing. since part of your application is written in sclang and part
would be written in (as an example) python, you need a way for the two
languages to talk to each other. the easiest way to do this is to run
python as a subprocess of sclang, e.g. "python myscript.py".unixCmd.


nathan

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

yorgos diapoulis
I am using pythonosc to send OSC data from SC3 to python and back to SC3.  IncrementalPCA is performed in python using scikit-learn.

2017-12-23 22:44 GMT+01:00 <[hidden email]>:
I'm fairly new to SC, but I know that it's not the language I want to do any serious data analysis in. My own interest is primarily in sonification, so that involves explicit non-sonic data being transformed into control signals.

My thinking at the moment is that it's best to separate the two cleanly - start with instruments that have clear, sonically relevant control parameters, and translate the data stream, whatever shape it comes in, into control signals. 

I'm using Python externally to process data streams, generating streams of sonic control signals, and sending OSC signals to SC.

Seems to work, but I'm aware that there is a ton of experience out there and I'm very interested in other people's ideas.

One thing external OSC has going for it is that it scales.

On Fri, Dec 22, 2017 at 7:13 PM <[hidden email]> wrote:
On 2017-12-22 04:27, [hidden email] wrote:
> Hi,
>
> Thank you for your reply.
>
> Can you elaborate more on .unixCmd?

hi kewping,

sure thing. since part of your application is written in sclang and part
would be written in (as an example) python, you need a way for the two
languages to talk to each other. the easiest way to do this is to run
python as a subprocess of sclang, e.g. "python myscript.py".unixCmd.


nathan

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/

Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

grahamsw
this is absolutely excellent. Thank you. 

On Wed, Dec 27, 2017 at 9:38 AM <[hidden email]> wrote:
I am using pythonosc to send OSC data from SC3 to python and back to SC3.  IncrementalPCA is performed in python using scikit-learn.

2017-12-23 22:44 GMT+01:00 <[hidden email]>:
I'm fairly new to SC, but I know that it's not the language I want to do any serious data analysis in. My own interest is primarily in sonification, so that involves explicit non-sonic data being transformed into control signals.

My thinking at the moment is that it's best to separate the two cleanly - start with instruments that have clear, sonically relevant control parameters, and translate the data stream, whatever shape it comes in, into control signals. 

I'm using Python externally to process data streams, generating streams of sonic control signals, and sending OSC signals to SC.

Seems to work, but I'm aware that there is a ton of experience out there and I'm very interested in other people's ideas.

One thing external OSC has going for it is that it scales.

On Fri, Dec 22, 2017 at 7:13 PM <[hidden email]> wrote:
On 2017-12-22 04:27, [hidden email] wrote:
> Hi,
>
> Thank you for your reply.
>
> Can you elaborate more on .unixCmd?

hi kewping,

sure thing. since part of your application is written in sclang and part
would be written in (as an example) python, you need a way for the two
languages to talk to each other. the easiest way to do this is to run
python as a subprocess of sclang, e.g. "python myscript.py".unixCmd.


nathan

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/

Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

kewping
Just sharing this link as is very good for understanding PCA from basic
mathematics. He discussed from basic mathematics terms (mean, variance,
co-variance, eigenvalue, eigenvector)

A Tutorial on Principal Component Analysis by Lindsay I Smith

https://klevas.mif.vu.lt/~tomukas/Knygos/principal_components.pdf





--
Sent from: http://new-supercollider-mailing-lists-forums-use-these.2681727.n2.nabble.com/SuperCollider-Users-New-Use-this-f2676391.html

_______________________________________________
sc-users mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-users/
search: http://www.listarc.bham.ac.uk/lists/sc-users/search/
Reply | Threaded
Open this post in threaded view
|

Re: Data set/ Dimensional Reduction / PCA (Principal Component Analysis)

crucialfelix

Yeah, PCA isn't that difficult. It can easily be done using the vector classes in MathLib. 

Probably less effort to just figure out how to implement the equation than trying to connect to an entirely separate programming environment just because somebody over there already figured it out.

On the other hand, there is a huge wealth of such things in python so having a working bridge would be nice.



On Fri, Jan 12, 2018 at 2:50 PM, <[hidden email]> wrote:
Just sharing this link as is very good for understanding PCA from basic
mathematics. He discussed from basic mathematics terms (mean, variance,
co-variance, eigenvalue, eigenvector)

A Tutorial on Principal Component Analysis by Lindsay I Smith

https://klevas.mif.vu.lt/~tomukas/Knygos/principal_components.pdf





--
Sent from: http://new-supercollider-mailing-lists-forums-use-these.2681727.n2.nabble.com/SuperCollider-Users-New-Use-this-f2676391.html