Commit da2f74fe authored by Somnath, Suhas's avatar Somnath, Suhas
Browse files

Merge branch 'cades_dev' of https://github.com/pycroscopy/pycroscopy into cades_dev_local

parents f94fc370 31c6b611
......@@ -87,7 +87,7 @@ Journal Papers using pycroscopy
 
4. `Direct Imaging of the Relaxation of Individual Ferroelectric Interfaces in a Tensile-Strained Film <http://onlinelibrary.wiley.com/doi/10.1002/aelm.201600508/full>`_ by L. Li et al.; Advanced Electronic Materials (2017), jupyter notebook `here 4 <http://nbviewer.jupyter.org/github/pycroscopy/pycroscopy/blob/master/jupyter_notebooks/BE_Processing.ipynb>`_
5. Feature extraction via similarity search: application to atom finding and denosing in electon and scanning probe microscopy imaging by S. Somnath et al.; submitted (2017), jupyter notebook `here 5 <http://nbviewer.jupyter.org/github/pycroscopy/pycroscopy/blob/master/jupyter_notebooks/Image_Cleaning_Atom_Finding.ipynb>`_
5. Feature extraction via similarity search: application to atom finding and denosing in electon and scanning probe microscopy imaging by S. Somnath et al.; under review at Advanced Structural and Chemical Imaging (2017), jupyter notebook `here 5 <http://nbviewer.jupyter.org/github/pycroscopy/pycroscopy/blob/master/jupyter_notebooks/Image_Cleaning_Atom_Finding.ipynb>`_
6. Many more coming soon....
......
......@@ -2,18 +2,17 @@
Roadmap / Milestones
--------------------
1. Sep 2017 end - Cleaned versions of the main modules (Analysis pending) + enough documentation for users and developers
2. Oct 2017 mid - Multi-node compute capability
1. Mid Sep - better BE notebook + visualization for users
2. Mid of Oct - Cleaned versions of the main modules (Analysis pending) + enough documentation for users and developers
3. End of Oct - Multi-node compute capability
New features
------------
Core development
~~~~~~~~~~~~~~~~
* A new class (pycro_data?) for simplifying the many data slicing, referencing operations on **main** datasets.
* Essentially, the goal is to turn the **main** datasets into powerful python objects that obviate the need for users to dig into ancillary datasets to slice, understand the datasets. Pycroscopy chooses to use a rather generalized representation of data at the cost of simplictiy. This object should bring back the simplicity of accessing the data.
* In the process of enabling greater insight into a dataset, this class would read and analyze ancillary datasets once and reuse this knowledge when the user requests another operation (that most likely also requires references to ancillary datasets etc. anyway).
* Nearly all the functionality has been implemented in hdf_utils and some in io_utils. This class can simply reuse these general functions.
* Generic visualizer in plot.lly / dash? that can use the pycrodata class
* Finish PycroDataset and test the many data slicing, referencing operations on **main** datasets.
* Need to be able to run a visualizer even on sliced data. What would this object be? (sliced Main data + vectors for each axis + axis labels ....). Perhaps the slice() function should return this object instead of a numpy array? As it stands, the additional information (for the visualizer) is not returned by the slice function.
* Generic visualizer in plot.lly / dash? that can use the PycroDataset class
* One suggestion is 2 (or more panes).
* Left hand side for positions
* 1D lines or 2D images
......@@ -22,7 +21,10 @@ Core development
* Right hand side for spectral
* 1D spectra or 2D images.
* Users will be asked to slice N-1 or N-2 spectral dimensions
* Simplify and demystify analyis / optimize. Use parallel_compute (joblib instead of multiprocessing)
* (re)Write at least one existing processing function as a class.
* Josh can redo fft filtering
* Rama can redo gIV bayesian inference
* Simplify and demystify analyis / optimize. Use parallel_compute instead of optimize and gues_methods and fit_methods
* multi-node computing capability in parallel_compute
* Data Generators
......@@ -54,53 +56,35 @@ Short tutorials on how to use pycroscopy
Longer examples (via specific scientific usecases)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Done:
* Data formatting in pycroscopy
* How to write a Translator
* How to write (back) to H5
* Spectral Unmixing with pycroscopy
* Basic introduction to loading data in pycroscopy
* Handling multidimensional (6D) datasets
* Visualizing data (interactively using widgets) (needs some tiny automation in the end)
* How to write your write your own parallel computing function using the process module
Pending:
* How to write your own analysis class
* How to write your write your own parallel computing function using the process module - add more documentation
* How to use the PycroDataset object
* A tour of the many functions in hdf_utils and io_utils since these functions need data to show / explain them.
* How to write your own analysis class based on the (to-be simplified) Model class
* pycroscopy pacakge organization - a short writeup on what is where and differences between the process / analyis submodules
Rama's (older and more applied / specific) tutorial goals
~~~~~~~~~~~~~~~~~~~~
1. Open a translated and fitted FORC-PFM file, and plot the SHO Fit from cycle k corresponding to voltage p, along with the raw spectrogram for that location and the SHO guess. Plot both real and imaginary, and do so for both on and off-field.
2. Continuing above, determine the average of the quality factor coming from cycles 1,3,4 for spatial points stored in vector b for the on-field part for a predetermined voltage range given by endpoints [e,f]. Compare the results with the SHO guess and fit for the quality factor.
3. After opening a h5 file containing results from a relaxation experiment, plot the response at a particular point and voltage, run exponential fitting and then store the results of the fit in the same h5 file using iohdf and/or numpy translators.
3. Done - After opening a h5 file containing results from a relaxation experiment, plot the response at a particular point and voltage, run exponential fitting and then store the results of the fit in the same h5 file using iohdf and/or numpy translators.
4. Take a FORC IV ESM dataset and break it up into forward and reverse branches, along with positive and negative branches. Do correlation analysis between PFM and IV for different branches and store the results in the file, and readily access them for plotting again.
5. A guide to using the model fitter for parallel fitting of numpy array-style datasets. This one can be merged with number
Documentation
-------------
* Switch from static examples to dynamic jupyter notebook like examples:
* http://scikit-image.org/docs/dev/auto_examples/
* http://scikit-learn.org/stable/auto_examples/index.html
* more complicated analyses - http://nipy.org/dipy/examples_index.html
* Done for existing documentation
* Work will be needed after examples are done
* Include examples in documentation
* Links to references for all functions and methods used in our workflows.
Formatting changes
------------------
* Fix remaining PEP8 problems
* Ensure code and documentation is standardized
* Switch to standard version formatting
* Classes and major Functions should check to see if the results already exist
Notebooks
---------
* Direct downloading of notebooks (ipynb an html)
* nbviewer?
* Host somewhere other than github?
* Investigate using JupyterLab
Testing
......
......@@ -132,15 +132,17 @@ dataset would be structured as:
| iN-1, j0 | iN-1, j1 | iN-1, j2 | .... | iN-1, jP-1 | iN-1, jP-1 |
+------------+------------+------------+--------+--------------+--------------+
* If the same voltage sweep were performed twice at each location, the data would be represented as N x 2 P.
The data is still saved as a long (2*P) 1D array at each location. The number of spectroscopic dimensions
would change from just ['Voltage'] to ['Voltage', 'Cycle'] where the second spectroscopic dimension would
account for repetitions of this bias sweep.
* **The spectroscopic data would be stored as it would be recorded as volt_0-cycle_0, volt_1-cycle_0.....
volt_P-1-cycle_0, volt_0-cycle_1.....volt_P-1-cycle-1. Just like the positions**
* Now, if the bias was swept thrice from -1 to +1V and then thrice again from -2 to 2V, the data bacomes
N x 2 * 3 P. The data now has two position dimensions (X, Y) and three spectrosocpic dimensions ['Voltage',
'Cycle', 'Step']. The data is still saved as a (P * 2 * 3) 1D array at each location.
* If the same voltage sweep were performed twice at each location, the data would be represented as N x 2 P.
The data is still saved as a long (2*P) 1D array at each location. The number of spectroscopic dimensions
would change from just ['Voltage'] to ['Voltage', 'Cycle'] where the second spectroscopic dimension would
account for repetitions of this bias sweep.
* **The spectroscopic data would be stored as it would be recorded as volt_0-cycle_0, volt_1-cycle_0.....
volt_P-1-cycle_0, volt_0-cycle_1.....volt_P-1-cycle-1. Just like the positions**
* Now, if the bias was swept thrice from -1 to +1V and then thrice again from -2 to 2V, the data bacomes
N x 2 * 3 P. The data now has two position dimensions (X, Y) and three spectrosocpic dimensions ['Voltage',
'Cycle', 'Step']. The data is still saved as a (P * 2 * 3) 1D array at each location.
- A collection of ``k`` chosen spectra would also be considered
``main`` datasets since the data is still structured as
......@@ -357,24 +359,18 @@ comfortably accomodates the pycroscopy format and offers several
advantageous features.
Information can be stored in HDF5 files in several ways:
* ``Datasets`` allow the storageo of data matricies and these are the
vessels used for storing the ``main``, ``ancillary``, and any extra data
matricies
* ``Datagroups`` are similar to folders in conventional
file systems and can be used to store any number of datasets or
* ``Datasets`` allow the storageo of data matricies and these are the vessels used for storing the ``main``,
``ancillary``, and any extra data matricies
* ``Datagroups`` are similar to folders in conventional file systems and can be used to store any number of datasets or
datagroups themselves
* ``Attributes`` are small pieces of
information, such as experimental or analytical parameters, that are
stored in key-value pairs in the same way as dictionaries in python.
Both datagroups and datasets can store attributes.
* While they are not
means to store data, ``Links`` or ``references`` can be used to
provide shortcuts and aliases to datasets and datagroups. This feature
is especially useful for avoiding duplication of datasets when two
``main`` datasets use the same ancillary datasets.
Among the `various
benefits <http://extremecomputingtraining.anl.gov/files/2015/03/HDF5-Intro-aug7-130.pdf>`__
* ``Attributes`` are small pieces of information, such as experimental or analytical parameters, that are stored in
key-value pairs in the same way as dictionaries in python. Both datagroups and datasets can store attributes.
* While they are not means to store data, ``Links`` or ``references`` can be used to provide shortcuts and aliases to
datasets and datagroups. This feature is especially useful for avoiding duplication of datasets when two ``main``
datasets use the same ancillary datasets.
Among the `various benefits <http://extremecomputingtraining.anl.gov/files/2015/03/HDF5-Intro-aug7-130.pdf>`__
that they offer, HDF5 files:
* are readily compatible with high-performance computing facilities
......@@ -479,9 +475,10 @@ time)
* ``labels`` - list of strings for the column names like ['Bias', 'Cycle']
* ``units`` – list of strings for units like ['V', ''].
Empty string for dimensionless quantities
Empty string for dimensionless quantities
* Optional attributes:
* Region references based on row names
Attributes
......@@ -557,7 +554,7 @@ Measurement data
- Datasets common to Channel\_000 and Channel\_001
- ....
- ...
Tool (analysis / processing)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
......
pycroscopy.analysis.be_loop_model
=================================
pycroscopy\.analysis\.be\_loop\_model
=====================================
.. automodule:: pycroscopy.analysis.be_loop_model
......@@ -9,15 +9,12 @@ pycroscopy.analysis.be_loop_model
.. rubric:: Classes
Classes
-------
.. autosummary::
.. autoclass:: BELoopModel
:members:
.. autoclass:: LoopOptimize
:members:
BELoopModel
LoopOptimize
......
pycroscopy.analysis.be_sho_model
================================
pycroscopy\.analysis\.be\_sho\_model
====================================
.. automodule:: pycroscopy.analysis.be_sho_model
......@@ -9,13 +9,11 @@ pycroscopy.analysis.be_sho_model
.. rubric:: Classes
Classes
-------
.. autosummary::
.. autoclass:: BESHOmodel
:members:
BESHOmodel
......
pycroscopy\.analysis\.fit\_methods
==================================
.. automodule:: pycroscopy.analysis.fit_methods
.. rubric:: Classes
.. autosummary::
BE_Fit_Methods
Fit_Methods
forc_iv_fit_methods
\ No newline at end of file
pycroscopy\.analysis\.guess\_methods
====================================
.. automodule:: pycroscopy.analysis.guess_methods
.. rubric:: Functions
.. autosummary::
r_square
.. rubric:: Classes
.. autosummary::
GuessMethods
\ No newline at end of file
pycroscopy.analysis.model
=========================
pycroscopy\.analysis\.model
===========================
.. automodule:: pycroscopy.analysis.model
......@@ -9,13 +9,11 @@ pycroscopy.analysis.model
.. rubric:: Classes
Classes
-------
.. autosummary::
.. autoclass:: Model
:members:
Model
......
pycroscopy\.analysis\.optimize
==============================
.. automodule:: pycroscopy.analysis.optimize
.. rubric:: Functions
.. autosummary::
targetFuncFit
targetFuncGuess
.. rubric:: Classes
.. autosummary::
Optimize
\ No newline at end of file
pycroscopy\.io\.be\_hdf\_utils
==============================
.. automodule:: pycroscopy.io.be_hdf_utils
.. rubric:: Functions
.. autosummary::
generateTestSpectroscopicData
getActiveUDVSsteps
getDataIndicesForUDVSstep
getForExcitWfm
getIndicesforPlotGroup
getSliceForExcWfm
getSpecSliceForUDVSstep
isReshapable
isSimpleDataset
maxReadPixels
reshapeToNsteps
reshapeToOneStep
\ No newline at end of file
pycroscopy\.io\.hdf\_utils
==========================
.. automodule:: pycroscopy.io.hdf_utils
.. rubric:: Functions
.. autosummary::
buildReducedSpec
calc_chunks
checkAndLinkAncillary
checkIfMain
check_for_old
copyAttributes
copyRegionRefs
copy_main_attributes
createRefFromIndices
create_empty_dataset
create_spec_inds_from_vals
findDataset
findH5group
getAuxData
getDataSet
getH5DsetRefs
getH5GroupRefs
getH5RegRefIndices
get_all_main
get_attr
get_attributes
get_data_descriptor
get_dimensionality
get_formatted_labels
get_sort_order
get_source_dataset
get_unit_values
linkRefAsAlias
linkRefs
link_as_main
print_tree
reducingRefCopy
reshape_from_Ndims
reshape_to_Ndims
simpleRefCopy
\ No newline at end of file
pycroscopy.io.io_hdf5
=====================
pycroscopy\.io\.io\_hdf5
========================
.. automodule:: pycroscopy.io.io_hdf5
......@@ -9,13 +9,11 @@ pycroscopy.io.io_hdf5
.. rubric:: Classes
Classes
-------
.. autosummary::
.. autoclass:: ioHDF5
:members:
ioHDF5
......
pycroscopy\.io\.io\_utils
=========================
.. automodule:: pycroscopy.io.io_utils
.. rubric:: Functions
.. autosummary::
check_dtype
check_ssh
complex_to_float
compound_to_scalar
getAvailableMem
getTimeStamp
realToComplex
realToCompound
recommendCores
transformToReal
transformToTargetType
uiGetFile
\ No newline at end of file
pycroscopy\.io\.microdata
=========================
.. automodule:: pycroscopy.io.microdata
.. rubric:: Classes
.. autosummary::
MicroData
MicroDataGroup
MicroDataset
\ No newline at end of file
pycroscopy.io.translators
=========================
pycroscopy\.io\.translators
===========================
.. automodule:: pycroscopy.io.translators
......
pycroscopy\.processing\.atom\_finding
=====================================
.. automodule:: pycroscopy.processing.atom_finding
.. rubric:: Functions
.. autosummary::
apply_binarization_filter
apply_binarization_filter_select
apply_find
apply_gaussian_corr_filter
apply_invert_filter
apply_select_channel
apply_wiener_filter
cluster_2d_oleg
cluster_2d_oleg_return_geo_center
cluster_into_atomic_columns
fun_2d_gaussian
return_img
return_pos
run_PCA_atoms
\ No newline at end of file
pycroscopy\.processing\.cluster
===============================
.. automodule:: pycroscopy.processing.cluster
.. rubric:: Functions
.. autosummary::
reorder_clusters
.. rubric:: Classes
.. autosummary::
Cluster
\ No newline at end of file
pycroscopy.processing.decomposition
===================================
pycroscopy\.processing\.decomposition
=====================================
.. automodule:: pycroscopy.processing.decomposition
......@@ -9,13 +9,11 @@ pycroscopy.processing.decomposition
.. rubric:: Classes
Classes
-------
.. autosummary::
.. autoclass:: Decomposition
:members:
Decomposition