Convert the source image file into a pycroscopy compatible hierarchical data format (HDF or .h5) file. This simple translation gives you access to the powerful data functions within pycroscopy
Convert the source image file into a pycroscopy compatible hierarchical data format (HDF or .h5) file. This simple translation gives you access to the powerful data functions within pycroscopy
#### H5 files:
#### H5 files:
* are like smart containers that can store matrices with data, folders to organize these datasets, images, metadata like experimental parameters, links or shortcuts to datasets, etc.
* are like smart containers that can store matrices with data, folders to organize these datasets, images, metadata like experimental parameters, links or shortcuts to datasets, etc.
* are readily compatible with high-performance computing facilities
* are readily compatible with high-performance computing facilities
* scale very efficiently from few kilobytes to several terabytes
* scale very efficiently from few kilobytes to several terabytes
* can be read and modified using any language including Python, Matlab, C/C++, Java, Fortran, Igor Pro, etc.
* can be read and modified using any language including Python, Matlab, C/C++, Java, Fortran, Igor Pro, etc.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
# Check if an HDF5 file with the chosen image already exists.
# Check if an HDF5 file with the chosen image already exists.
print('HDF5 file with Raw_Data found. No need to translate.')
print('HDF5 file with Raw_Data found. No need to translate.')
exceptKeyError:
exceptKeyError:
print('Raw Data not found.')
print('Raw Data not found.')
else:
else:
print('No HDF5 file found.')
print('No HDF5 file found.')
ifneed_translation:
ifneed_translation:
# Initialize the Image Translator
# Initialize the Image Translator
tl=px.ImageTranslator()
tl=px.ImageTranslator()
# create an H5 file that has the image information in it and get the reference to the dataset
# create an H5 file that has the image information in it and get the reference to the dataset
h5_raw=tl.translate(image_path)
h5_raw=tl.translate(image_path)
# create a reference to the file
# create a reference to the file
h5_file=h5_raw.file
h5_file=h5_raw.file
print('HDF5 file is located at {}.'.format(h5_file.filename))
print('HDF5 file is located at {}.'.format(h5_file.filename))
```
```
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
### Inspect the contents of this h5 data file
### Inspect the contents of this h5 data file
The file contents are stored in a tree structure, just like files on a contemporary computer.
The file contents are stored in a tree structure, just like files on a contemporary computer.
The data is stored as a 2D matrix (position, spectroscopic value) regardless of the dimensionality of the data.
The data is stored as a 2D matrix (position, spectroscopic value) regardless of the dimensionality of the data.
In the case of these 2D images, the data is stored as a N x 1 dataset
In the case of these 2D images, the data is stored as a N x 1 dataset
The main dataset is always accompanied by four ancillary datasets that explain the position and spectroscopic value of any given element in the dataset.
The main dataset is always accompanied by four ancillary datasets that explain the position and spectroscopic value of any given element in the dataset.
In the case of the 2d images, the positions will be arranged as row0-col0, row0-col1.... row0-colN, row1-col0....
In the case of the 2d images, the positions will be arranged as row0-col0, row0-col1.... row0-colN, row1-col0....
The spectroscopic information is trivial since the data at any given pixel is just a scalar value
The spectroscopic information is trivial since the data at any given pixel is just a scalar value
## Performing Singular Value Decompostion (SVD) on the windowed data
## Performing Singular Value Decompostion (SVD) on the windowed data
SVD decomposes data (arranged as position x value) into a sequence of orthogonal components arranged in descending order of variance. The first component contains the most significant trend in the data. The second component contains the next most significant trend orthogonal to all previous components (just the first component). Each component consists of the trend itself (eigenvector), the spatial variaion of this trend (eigenvalues), and the variance (statistical importance) of the component.
SVD decomposes data (arranged as position x value) into a sequence of orthogonal components arranged in descending order of variance. The first component contains the most significant trend in the data. The second component contains the next most significant trend orthogonal to all previous components (just the first component). Each component consists of the trend itself (eigenvector), the spatial variaion of this trend (eigenvalues), and the variance (statistical importance) of the component.
Since the data consists of the large sequence of small windows, SVD essentially compares every single window with every other window to find statistically significant trends in the image
Since the data consists of the large sequence of small windows, SVD essentially compares every single window with every other window to find statistically significant trends in the image
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
# check to make sure number of components is correct:
# check to make sure number of components is correct:
The plot below shows the variance or statistical significance of the SVD components. The first few components contain the most significant information while the last few components mainly contain noise.
The plot below shows the variance or statistical significance of the SVD components. The first few components contain the most significant information while the last few components mainly contain noise.
Note also that the plot below is a log-log plot. The importance of each subsequent component drops exponentially.
Note also that the plot below is a log-log plot. The importance of each subsequent component drops exponentially.
%% Cell type:code id: tags:
%% Cell type:code id: tags:
``` python
``` python
fig_S,ax_S=px.plot_utils.plotScree(h5_S[()]);
fig_S,ax_S=px.plot_utils.plotScree(h5_S[()]);
```
```
%% Cell type:markdown id: tags:
%% Cell type:markdown id: tags:
#### V (Eigenvectors or end-members)
#### V (Eigenvectors or end-members)
The V dataset contains the end members for each component
The V dataset contains the end members for each component
We will attempt to find the positions and the identities of atoms in the image now
We will attempt to find the positions and the identities of atoms in the image now
## Perform clustering on the dataset
## Perform clustering on the dataset
Clustering divides data into k clusters such that the variance within each cluster is minimized.<br>
Clustering divides data into k clusters such that the variance within each cluster is minimized.<br>
Here, we will be performing k-means clustering on a set of components in the U matrix from SVD.<br>
Here, we will be performing k-means clustering on a set of components in the U matrix from SVD.<br>
We want a large enough number of clusters so that K-means identifies fine nuances in the data. At the same time, we want to minimize computational time by reducing the number of clusters. We recommend 32 - 64 clusters.
We want a large enough number of clusters so that K-means identifies fine nuances in the data. At the same time, we want to minimize computational time by reducing the number of clusters. We recommend 32 - 64 clusters.
Here, we will interactively identify N windows, each centered on a distinct class / kind of atom.
Here, we will interactively identify N windows, each centered on a distinct class / kind of atom.
Use the coarse and fine positions sliders to center the window onto target atoms. Click the "Set as motif" button to add this window to the list of patterns we will search for in the next step. Avoid duplicates.
Use the coarse and fine positions sliders to center the window onto target atoms. Click the "Set as motif" button to add this window to the list of patterns we will search for in the next step. Avoid duplicates.
Note: If a pair of motifs are always matching for the same set of atoms, perhaps this may be a duplicate motif. Alternatively, if these motifs do indeed identify distinct classes of atoms, consider:
Note: If a pair of motifs are always matching for the same set of atoms, perhaps this may be a duplicate motif. Alternatively, if these motifs do indeed identify distinct classes of atoms, consider:
* clustering again with a different set of SVD components
* clustering again with a different set of SVD components
* increasing the number of clusters
* increasing the number of clusters
* Choosing a different fft mode ('data+fft' for better identify subtle but important variations) before performing windowing on the data
* Choosing a different fft mode ('data+fft' for better identify subtle but important variations) before performing windowing on the data
We do this by thresholding the matching scores such that a score beyond the threshold is set to 1 and all other values are set to 0.
We do this by thresholding the matching scores such that a score beyond the threshold is set to 1 and all other values are set to 0.
The goal is to set the thresholds such that we avoid overlaps between two clusters and also shrink the blobs such that they are only centered over a single atom wherever possible.
The goal is to set the thresholds such that we avoid overlaps between two clusters and also shrink the blobs such that they are only centered over a single atom wherever possible.
Use the sliders below to interactively set the threshold values
Use the sliders below to interactively set the threshold values
