Lab Skills: Converting file formats

Developed by E. Ayars

It is all too common in scientific research to have to deal with equipment that for one reason or another produces data files in an inconvenient format. The example provided here is a Rigaku X-Ray diffractometer, an extremely precise high-quality bit of equipment which unfortunately dates from the era of dot-matrix printers. It insists on producing data files with sequential measurements broken into multiple rows, a format which is useful for line printers but not for anything else. This exercise gives the student real-world experience in converting this data into a more useful format.
Subject Area Experimental Labs
Levels First Year and Beyond the First Year
Available Implementation Python
Learning Objectives
Students who complete this exercise will * be able to write pseudocode to plan an approach to reading (and converting) data files. (**Exercise 1**) * be able to extract data from a file header that they can then use to interpret the rest of the file. (**Exercises 1, 2, 4**) * be able to extract and re-configure data from the bulk of the file. (**Exercises 1, 2**) * be able to save the converted data in a useful format for further use. (**Exercises 2, 3**) * be able to produce graphs of data from a file. (**Exercise 4**)
Time to Complete 60 min
It is all too common in scientific research to have to deal with equipment that for one reason or another produces data files in an inconvenient format. The example provided here is a Rigaku X-Ray diffractometer, an extremely precise high-quality bit of equipment which unfortunately dates from the era of dot-matrix printers. It insists on producing data files with sequential measurements broken into multiple rows, a format which is useful for line printers but not for anything else. This exercise gives the student real-world experience in converting this data into a more useful format. ## Exercise 1: Planning Here's the format for the ".MDI" files as they now come to the computer: ``` 02-24-11 @17:07 DIF Si test 3 3.0 0.05 1.0 CU 1.54056 69.95 1340 104 109 105 110 126 119 97 103 114 129 119 130 122 106 117 128 129 122 141 128 119 125 128 135 137 139 142 126 153 143 125 151 ... ``` The first line contains the date and time and a user-supplied description of what's in the file. The second line displays, in order, the start angle (3.0°), the angle step-size (0.05°), source information (1.0 CU 1.54056) the final angle (69.95°), and the number of points (1340). The third line begins the data, with successive values running across the rows. So in that sample set, the actual data we want would be (in a useful two-column form) | $\theta$ | count | | -------- | ----- | | 3.00 | 104 | | 3.05 | 109 | | 3.10 | 105 | | ... | ... | | 3.35 | 103 | | 3.40 | 114 | | ... | ... | Write pseudo-code that explains, step by step, how you will retrieve the information from this file and put it into the desired format. ## Exercise 2: Programming Write a program that implements the plan from Exercise 1. Your program should save the data as a text file in the new format. ## Exercise 3: Getting Fancy Modify your program from Exercise 2 to be more generally useful. For example, does it currently work on one hard-coded filename? If so, fix it so that it asks the user for a filename, or better yet retrieves that filename from the command that invoked the program in the first place. ## Exercise 4: Analysis As long as you're doing things with numbers anyway, make the program display a graph of the data. After all, that's likely why you're interested in the data in the first place! Find the angles of maximum reflection for one of the data files. Using the information in the header file, can you determine the spacing between crystal layers in this X-Ray Diffraction data?

Download Options

Share a Variation

Did you have to edit this material to fit your needs? Share your changes by Creating a Variation

Credits and Licensing

E. Ayars, "Lab Skills: Converting file formats," Published in the PICUP Collection, August 2016.

The instructor materials are ©2016 E. Ayars.

The exercises are released under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license