Parsers Example

This example will demonstrate how diffpy.utils lets us easily process and serialize files. Using the parsers module, we can load file data into simple and easy-to-work-with Python objects.

  1. To begin, unzip parser_data and take a look at data.txt. This is a fairly standard format for 1D powder diffraction data. Our goal will be to extract the data, and the parameters listed in the header, from this file and load it into our program.

  2. To get the data table, we will use the loadData function. The default behavior of this function is to find and extract a data table from a file.

from diffpy.utils.parsers.loaddata import loadData
data_table = loadData('<PATH to data.txt>')

While this will work with most datasets, on our data.txt file, we got a ValueError. The reason for this is due to the comments $ Phase Transition Near This Temperature Range and --> Note Significant Jump in Rw <-- embedded within the dataset. To fix this, try using the comments parameter.

data_table = loadData('<PATH to data.txt>', comments=['$', '-->'])

This parameter tells loadData that any lines beginning with $ and --> are just comments and more entries in our data table may follow.

Here are a few other parameters to test out:

  • delimiter=',': Look for a comma-separated data table. Useful for csv file types. However, since data.txt is whitespace separated, running

loadData('<PATH to data.txt>', comments=['$', '-->'], delimiter=',')
returns an empty list.

  • minrows=50: Only look for data tables with at least 50 rows. Since our data table has much less than that many rows, running

loadData('<PATH to data.txt>', comments=['$', '-->'], minrows=50)
returns an empty list.

  • usecols=[0, 3]: Only return the 0th and 3rd columns (zero-indexed) of the data table. For data.txt, this corresponds to the temperature and rw columns.

loadData('<PATH to data.txt>', comments=['$', '-->'], usecols=[0, 3])

  1. Next, to get the header information, we can again use loadData, but this time with the headers parameter enabled.

hdata = loadData('<PATH to data.txt>', comments=['$', '-->'], headers=True)

  1. Rather than working with separate data_table and hdata objects, it may be easier to combine them into a single dictionary. We can do so using the serialize_data function.

from diffpy.utils.parsers.loaddata import serialize_data
file_data = serialize_data('<PATH to data.txt', hdata, data_table)
# File data is a dictionary with a single key
# The key is the file name (in our case, 'data.txt')
# The entry is a dictionary containing data from hdata and data_table
data_dict = file_data['data.txt']

This dictionary data_dict contains all entries in hdata and an additional entry named data table containing data_table.

here_is_the_data_table = data_dict['data table']

There is also an option to name columns in the data table and save those columns as entries instead.

data_table_column_names = ['temperature', 'scale', 'stretch', 'rw']  # names of the columns in data.txt
file_data = serialize_data('<PATH to data.txt>', hdata, data_table, dt_colnames=data_table_column_names)
data_dict = file_data['data.txt']

Now we can extract specific data table columns from the dictionary.

  1. When we are done working with the data, we can store it on disk for later use. This can also be done using the serialize_data function with an additional serial_file parameter.

  parsed_file_data = serialize_data('<PATH to data.txt>', hdata, data_table, serial_file='<PATH to serialfile.json>')

The returned value, ``parsed_file_data``, is the dictionary we just added to ``serialfile.json``.
To extract the data from the serial file, we use ``deserialize_data``.

from diffpy.utils.parsers.serialization import deserialize_data
parsed_file_data = deserialize_data('<PATH to serialdata.json>')

  1. Finally, serialize_data allows us to store data from multiple text files in a single serial file. For one last bit of practice, we will extract and add the data from moredata.txt into the same serialdata.json file.

data_table = loadData('<PATH to moredata.txt>')
hdata = loadData('<PATH to moredata.txt>', headers=True)
serialize_data('<PATH to moredata.txt>', hdata, data_table, serial_file='<PATH to serialdata.json>')

The serial file serialfile.json should now contain two entries: data.txt and moredata.txt. The data from each file can be accessed using

serial_data = deserialize_data('<PATH to serialdata.json>')
data_txt_data = serial_data['data.txt']  # Access data.txt data
moredata_txt_data = serial_data['moredata.txt']  # Access moredata.txt data

For more information, check out the documentation of the parsers module. b