EC703 Topics in AI: ICA

• Syllabus (Get the postscript file)

• If you are a beginner in using MATLAB, click here to get "Using Matlab".

• Announcement

  • Please read the first two articles (1 and 2 in the reading list) before you come to my class on the 30th of August.

  • No class on the 6th of September.

  • The first homework assignment has been posted.

  • The lecture (9/11) will cover the articles [1,2].

  • Lectures (9/13, 9/18) will cover the articles [4,7,6].

  • The 3rd homework assignment has been posted.

  • As of the 4th of October, class will begin at 9:00 am (instead of 9:30 am). Please be in the classroom by 8:55 am.
    Please read Chapter 2 (and possibly Chapter 3).

  • Three more articles [9-11] have been added in the reading list.
    The article [9] will be discussed in 10/9 lecture.
    Lectures on 10/11 and 10/16 will talk about [10,11].

  • New homework assignments (hwk 6 and hwk 7) have been posted. Stay tuned!

  • A new article has been added in the reading list.

  • A final term project has been posted here.

• Handouts

  • Introduction ( download )

  • Higher-order statistics ( download )

  • Mathematical preliminaries for source separation ( download )

• Homework Assignments

  • Hwk 1
    

    • Prove the first (CP1) and last (CP7) properties of cumulants.

    • Due September 11 (Tue)

  • Hwk 2

    • Computer exercise: Implement FOBI.

    • Get the data set ( download ).
      The observation data consists of mixtures of 5 different sounds with 10000 samples.
      All of them were sampled at 8kHz. Try "soundsc(x(1,:))" to hear the sound.
      Your report should include: (1) summary of the algorithm; (2) plots of 5 recovered signals; (3) your source code.

    • You can download the mixing matrix and the original source signals

    • Due September 18 (Tue)

  • Hwk 3

    • Computer exercise: Implement AMUSE. Use the same data set as hwk 2.

    • Due September 25 (Tue)

  • Hwk 4

    • Prove that the mean of score function is zero.

    • Due October 11 (Thu)

  • Hwk 5 (Corresponding to Midterm 1)

    • Detailed derivation of EM for PPCA.

    • Computer exercise: Implement PPCA.

    • Your report must include algorithm derivation and implementation.
      You also have to include at least one example to convince me your implementation is working.
      More experimental results, more credit.

    • This assignment will take up 10% of your grade.

    • Due 10/25 (Thu).

  • Hwk 6 (Corresponding to Midterm 2)

    • Detailed derivation of EM for Mixtures of PCA [11]

    • Computer exercise: Implement Mixtures of PCA

    • Your report must include algorithm derivation and implementation.
      You have to include a toy example (you generate a data set by yourself)
      and an advanced example using a data set from UCI Repository .
      (Choose whatever you like, but you might be interested in Wine Data)

    • This assignment will take up 10% of your grade.

    • Due 11/6 (Tue).

  • Hwk 7 (Corresponding to Midterm 3)

    • Detailed algorithm derivation of GTM [14]

    • Computer exercise: Implement GTM

    • Your report must include algorithm derivation and implementation.
      The Oil Flow data set is available from Bishop's website.
      (if you read his paper carefully, you can find where you can get the data set)

    • This assignment will take up 10% of your grade.

    • Due 11/16 (Thu).

  • Hwk 8 (Corresponding to Final Term Project)

    • Download the paper on EM-ICA.

    • Detailed algorithm derivation of EM-ICA.

    • Implement EM-ICA with a toy example.

    • Show me the performance of your code with realistic data.

    • Due 12/17 (Mon) Noon.

• Reading List

  1. J. -F. Cardoso,
     "Source Separation Using Higher-order Moments,"
     in Proc. ICASSP, 1989. ( download )
     FOBI algorithm

  2. L. Tong, V. Soon, Y. F. Huang, and R. Liu,
     "Indeterminacy and identifiability of blind identification,"
     IEEE Trans. CAS, vol. 38, pp. 499-509, 1991. ( download )
     Identifiability, EFOBI and AMUSE algorithms

     (http://www.ics.uci.edu/pub/machine-learning-databases/) You also have to include one toy example and to convince me your implementation is working.
     More experimental results, more credit.

  3. L. Tong, Y. Inouye, and R. Liu,
     "Waveform-preserving blind estimation of multiple independent sources,"
     IEEE Trans. SP, vol. 41, pp. 2461-2470, 1993. ( download )
     Cumulants-basded source separation

  4. J. -F. Cardoso and A. Souloumiac,
     "Blind beamforming for Non-Gaussian signals,"
     IEE Proceedings-F, vol. 140, no. 6, pp. 362-370, 1993. ( download )
     JADE algorithm

  5. J. -F. Cardoso,
     "Blind signal separation: Statistical principles,"
     Proc. IEEE, vol. 86, no. 10, pp. 2009-2025, 1998 ( download )
     Tutorial on source separation

  6. J. -F. Cardoso,
     "Higher-order contrasts for ICA,"
     Neural Computation, vol. 11, no. 1, p. 157-192, 1999 ( download )
     Cumulants-based contrasts

  7. A. Belouchrani, K. Abed-Merain, J. -F. Cardoso, and E. Moulines,
     "A blind source separation technique using second order statistics,"
     IEEE Trans. SP, vol. 45, pp. 434-444, 1997. ( download )
     SOBI algorithm

  8. S. Amari and A. Cichocki,
     "Adaptive blind signal processing - Neural network approaches,"
     Proc. IEEE, vol. 86, pp. 2026-2048, 1998. ( download )
     Tutorial on source separation

  9. D. MacKay,
     "Maximum likelihood and covariant algorithms for ICA,"
     Technical Report, Cavendish Lab, Cambridge, 1996. ( download )
     ML ICA

  10. M. E. Tipping and C. M. Bishop,
      "Probabilistic Principal Component Analysis,"
      Journal of the Royal Statistical Society B, vol. 61, no.3, pp. 611-622, 1999. ( download )
      EM algorithm for PCA

  11. M. E. Tipping and C. M. Bishop,
      "Mixtures of Probabilistic Principal Component Analyzers,"
      Neural Computation, vol. 11, no. 2, pp. 443-482, 1999. ( download )
      EM algorithm for mixtures of PCA

  12. Z. Ghahramnani and G. Hinton,
      "The EM algorithm for mixtures of factor analyzers,"
      Technical Report, 1996. ( download )
      EM algorithm for mixtures of FA

  13. S. Roweis,
      "EM algorithms for PCA and SPCA,"
      Advances in Neural Information Processing Systems 10 (NIPS'97), pp. 626-632. ( download )
      EM algorithm for PCA

  14. C. M. Bishop, M. Svensen, and C. K. I. Williams,
      "GTM: The Generative Topographic Mapping,"
      Neural Computation, vol. 10, pp. 215-234, 1998. ( download )
      GTM

  15. Z. Ghahramani and G. E. Hinton,
      "Parameter estimation for linear dynamical systems,"
      Technical Report. ( download )
      Linear Dynamic Systems

  16. T. W. Lee, M. Girolami, A. Bell, and T. Sejnowski,
      "A unifying information-theoretic framework for ICA,"
      Int. J. Mathematical and Computer Modeling. ( download )
      Information-theoretic learning for ICA

  17. S. Choi, A. Cichocki, and S. Amari,
      "Flexible independent component analysis,"
      Journal of VLSI Signal Processing, 2000. ( download )
      Flexible ICA

  18. A. Hyvarinen and E. Oja,
      "A fast fixed point algorithm for ICA,"
      Neural Computation, 1997. ( download )
      Fixed point ICA