README.md 2.09 KB
Newer Older
DEVARAKONDAR email's avatar
DEVARAKONDAR email committed
1
The File-Level Metadata Data (FLMD) parser is a lightweight python script that can produce a summary of fields based on column headers within a Comma-separated Values (CSV). The script enables a way to extract certain metadata fields from CSV data files. An optional configuration file can be used to select specific column headers to extract a smaller subset of values. The script can be used on a single file, or it can be given a directory to process groups of files in bulk. The  FLMD parser outputs the metadata is a JSON and can be configured to update the source file or produce an external companion file.
DEVARAKONDAR email's avatar
DEVARAKONDAR email committed
2
3


Crow, Michael C's avatar
Crow, Michael C committed
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# ESS-DIVE-META
Extract file-level metadata from csv files in a directory.

# Requirements
- Python 3.5+

# Installation
Use the package manager [pip](https://pip.pypa.io/en/stable/) to install the following packages.
```bash
python3 -m pip install numpy
python3 -m pip install pandas
```

# Setup
In the same directory with the script, create a folder, and put csv files there.

    .
    ├── ess-dive-meta.py
    ├── csv
    │   └── NWT_wildfires_biophysical_2016.csv
    └── config.csv
    
# Usage
```bash
# help documentation
Crow, Michael C's avatar
Crow, Michael C committed
29
python3 ess-dive-meta.py --help
Crow, Michael C's avatar
Crow, Michael C committed
30
31
32
33
34
35

# complete summary
python3 ess-dive-meta.py --directory example/

# only specified within config.csv
python3 ess-dive-meta.py --directory example/ --config config.csv
Crow, Michael C's avatar
Crow, Michael C committed
36
37
38
39
40
41

# only specified within config.csv and on a single file
python3 ess-dive-meta.py --file example/NWT_wildfires_biophysical_2016.csv --config config.csv

# shorthand
python3 ess-dive-meta.py -f example/NWT_wildfires_biophysical_2016.csv -g config.csv
Crow, Michael C's avatar
Crow, Michael C committed
42
43
44
45
46
47
48
```

# Config File Format
Simple csv file with only the names of the column headers you wish to extract.

Example:

Crow, Michael C's avatar
Crow, Michael C committed
49
50
    site,latitude,longitude,time,ecosystem 

51
# Citation
DEVARAKONDAR email's avatar
DEVARAKONDAR email committed
52
McNelis, J., Crow, M., and Devarakonda, R., ESS-DIVE File Level Metadata Extractor. Computer Software. https://code.ornl.gov/ngee-arctic/ess-dive-meta. 01 Oct. 2020. Web. doi:10.11578/dc.20201103.5.
53

Crow, Michael C's avatar
Crow, Michael C committed
54
55
56
57
58
59
# License
[UT-Battelle](LICENSE.txt)
|
[3-Clause BSD License](https://opensource.org/licenses/BSD-3-Clause)

Copyright 2020, UT-Battelle, LLC