plot_microdata_example.ipynb 6.73 KB
Newer Older
1
2
3
{
  "cells": [
    {
Unknown's avatar
Unknown committed
4
      "cell_type": "code",
5
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
6
7
8
      "metadata": {
        "collapsed": false
      },
9
      "outputs": [],
10
11
      "source": [
        "%matplotlib inline"
Unknown's avatar
Unknown committed
12
      ]
13
14
    },
    {
Unknown's avatar
Unknown committed
15
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
16
      "metadata": {},
17
18
      "source": [
        "\n# Writing to hdf5 using the Microdata objects\n\n\n\n\n\n"
Unknown's avatar
Unknown committed
19
      ]
20
21
    },
    {
Unknown's avatar
Unknown committed
22
      "cell_type": "code",
23
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
24
25
26
      "metadata": {
        "collapsed": false
      },
27
      "outputs": [],
28
      "source": [
29
        "# Code source: Chris Smith -- cq6@ornl.gov\n# Liscense: MIT\n\nimport os\nimport numpy as np\nimport pycroscopy as px"
Unknown's avatar
Unknown committed
30
      ]
31
32
    },
    {
Unknown's avatar
Unknown committed
33
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
34
      "metadata": {},
35
36
      "source": [
        "Create some MicroDatasets and MicroDataGroups that will be written to the file.\nWith h5py, groups and datasets must be created from the top down,\nbut the Microdata objects allow us to build them in any order and link them later.\n\n"
Unknown's avatar
Unknown committed
37
      ]
38
39
    },
    {
Unknown's avatar
Unknown committed
40
      "cell_type": "code",
41
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
42
43
44
      "metadata": {
        "collapsed": false
      },
45
      "outputs": [],
46
47
      "source": [
        "# First create some data\ndata1 = np.random.rand(5, 7)"
Unknown's avatar
Unknown committed
48
      ]
49
50
    },
    {
Unknown's avatar
Unknown committed
51
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
52
      "metadata": {},
53
54
      "source": [
        "Now use the array to build the dataset.  This dataset will live\ndirectly under the root of the file.  The MicroDataset class also implements the\ncompression and chunking parameters from h5py.Dataset.\n\n"
Unknown's avatar
Unknown committed
55
      ]
56
57
    },
    {
Unknown's avatar
Unknown committed
58
      "cell_type": "code",
59
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
60
61
62
      "metadata": {
        "collapsed": false
      },
63
      "outputs": [],
64
65
      "source": [
        "ds_main = px.MicroDataset('Main_Data', data=data1, parent='/')"
Unknown's avatar
Unknown committed
66
      ]
67
68
    },
    {
Unknown's avatar
Unknown committed
69
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
70
      "metadata": {},
71
72
      "source": [
        "We can also create an empty dataset and write the values in later\nWith this method, it is neccessary to specify the dtype and maxshape kwarg parameters.\n\n"
Unknown's avatar
Unknown committed
73
      ]
74
75
    },
    {
Unknown's avatar
Unknown committed
76
      "cell_type": "code",
77
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
78
79
80
      "metadata": {
        "collapsed": false
      },
81
      "outputs": [],
82
83
      "source": [
        "ds_empty = px.MicroDataset('Empty_Data', data=[], dtype=np.float32, maxshape=[7, 5, 3])"
Unknown's avatar
Unknown committed
84
      ]
85
86
    },
    {
Unknown's avatar
Unknown committed
87
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
88
      "metadata": {},
89
90
      "source": [
        "We can also create groups and add other MicroData objects as children.\nIf the group's parent is not given, it will be set to root.\n\n"
Unknown's avatar
Unknown committed
91
      ]
92
93
    },
    {
Unknown's avatar
Unknown committed
94
      "cell_type": "code",
95
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
96
97
98
      "metadata": {
        "collapsed": false
      },
99
      "outputs": [],
100
101
      "source": [
        "data_group = px.MicroDataGroup('Data_Group', parent='/')\n\nroot_group = px.MicroDataGroup('/')\n\n# After creating the group, we then add an existing object as its child.\ndata_group.addChildren([ds_empty])\nroot_group.addChildren([ds_main, data_group])"
Unknown's avatar
Unknown committed
102
      ]
103
104
    },
    {
Unknown's avatar
Unknown committed
105
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
106
      "metadata": {},
107
108
      "source": [
        "The showTree method allows us to view the data structure before the hdf5 file is\ncreated.\n\n"
Unknown's avatar
Unknown committed
109
      ]
110
111
    },
    {
Unknown's avatar
Unknown committed
112
      "cell_type": "code",
113
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
114
115
116
      "metadata": {
        "collapsed": false
      },
117
      "outputs": [],
118
119
      "source": [
        "root_group.showTree()"
Unknown's avatar
Unknown committed
120
      ]
121
122
    },
    {
Unknown's avatar
Unknown committed
123
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
124
      "metadata": {},
125
126
      "source": [
        "Now that we have created the objects, we can write them to an hdf5 file\n\n"
Unknown's avatar
Unknown committed
127
      ]
128
129
    },
    {
Unknown's avatar
Unknown committed
130
      "cell_type": "code",
131
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
132
133
134
      "metadata": {
        "collapsed": false
      },
135
      "outputs": [],
136
137
      "source": [
        "# First we specify the path to the file\nh5_path = 'microdata_test.h5'\n\n# Then we use the ioHDF5 class to build the file from our objects.\nhdf = px.ioHDF5(h5_path)"
Unknown's avatar
Unknown committed
138
      ]
139
140
    },
    {
Unknown's avatar
Unknown committed
141
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
142
      "metadata": {},
143
144
      "source": [
        "The writeData method builds the hdf5 file using the structure defined by the\nMicroData objects.  It returns a list of references to all h5py objects in the\nnew file.\n\n"
Unknown's avatar
Unknown committed
145
      ]
146
147
    },
    {
Unknown's avatar
Unknown committed
148
      "cell_type": "code",
149
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
150
151
152
      "metadata": {
        "collapsed": false
      },
153
      "outputs": [],
154
155
      "source": [
        "h5_refs = hdf.writeData(root_group, print_log=True)\n\n# We can use these references to get the h5py dataset and group objects\nh5_main = px.io.hdf_utils.getH5DsetRefs(['Main_Data'], h5_refs)[0]\nh5_empty = px.io.hdf_utils.getH5DsetRefs(['Empty_Data'], h5_refs)[0]"
Unknown's avatar
Unknown committed
156
      ]
157
158
    },
    {
Unknown's avatar
Unknown committed
159
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
160
      "metadata": {},
161
162
      "source": [
        "Compare the data in our dataset to the original\n\n"
Unknown's avatar
Unknown committed
163
      ]
164
165
    },
    {
Unknown's avatar
Unknown committed
166
      "cell_type": "code",
167
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
168
169
170
      "metadata": {
        "collapsed": false
      },
171
      "outputs": [],
172
173
      "source": [
        "print(np.allclose(h5_main[()], data1))"
Unknown's avatar
Unknown committed
174
      ]
175
176
    },
    {
Unknown's avatar
Unknown committed
177
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
178
      "metadata": {},
179
180
      "source": [
        "As mentioned above, we can now write to the Empty_Data object\n\n"
Unknown's avatar
Unknown committed
181
      ]
182
183
    },
    {
Unknown's avatar
Unknown committed
184
      "cell_type": "code",
185
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
186
187
188
      "metadata": {
        "collapsed": false
      },
189
      "outputs": [],
190
191
      "source": [
        "data2 = np.random.rand(*h5_empty.shape)\nh5_empty[:] = data2[:]"
Unknown's avatar
Unknown committed
192
      ]
193
194
    },
    {
Unknown's avatar
Unknown committed
195
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
196
      "metadata": {},
197
198
      "source": [
        "Now that we are using h5py objects, we must use flush to write the data to file\nafter it has been altered.\nWe need the file object to do this.  It can be accessed as an attribute of the\nhdf object.\n\n"
Unknown's avatar
Unknown committed
199
      ]
200
201
    },
    {
Unknown's avatar
Unknown committed
202
      "cell_type": "code",
203
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
204
205
206
      "metadata": {
        "collapsed": false
      },
207
      "outputs": [],
208
209
      "source": [
        "h5_file = hdf.file\nh5_file.flush()"
Unknown's avatar
Unknown committed
210
      ]
211
212
    },
    {
Unknown's avatar
Unknown committed
213
      "cell_type": "markdown",
Unknown's avatar
Unknown committed
214
      "metadata": {},
215
216
      "source": [
        "Now that we are done, we should close the file so that it can be accessed elsewhere.\n\n"
Unknown's avatar
Unknown committed
217
      ]
218
219
    },
    {
Unknown's avatar
Unknown committed
220
      "cell_type": "code",
221
      "execution_count": null,
CompPhysChris's avatar
CompPhysChris committed
222
223
224
      "metadata": {
        "collapsed": false
      },
225
      "outputs": [],
226
      "source": [
227
        "h5_file.close()\nos.remove(h5_path)"
Unknown's avatar
Unknown committed
228
      ]
229
    }
230
  ],
Unknown's avatar
Unknown committed
231
  "metadata": {
232
233
234
235
236
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
Unknown's avatar
Unknown committed
237
238
    "language_info": {
      "codemirror_mode": {
239
240
        "name": "ipython",
        "version": 3
Unknown's avatar
Unknown committed
241
242
      },
      "file_extension": ".py",
243
      "mimetype": "text/x-python",
Unknown's avatar
Unknown committed
244
      "name": "python",
245
246
247
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.3"
Unknown's avatar
Unknown committed
248
    }
249
250
251
  },
  "nbformat": 4,
  "nbformat_minor": 0
252
}