Data loaders

As emphasized in Data format section, different experimental setups return different file formats. To handle those discrepancies, data_loaders module implements objects that take care of a proper data extraction and returns data_loaders.Dataset, recognizable by the rest of the package.

It provides the Dataloader class as a base structure for different sources (beamlines, electron analyzers, labs) and a number of specific Dataloader dedicated to various ARPES beamlines around the world (full list is shown in a table below).

Principles of the data import in piva are presented on a simple scheme below.

If the file format is not yet supported, one needs to implement a custom Dataloader and make it accessible for piva. Detailed guidelines on how to do it can be found in section “Writing custom Dataloader”.

Implemented Dataloaders

Table below shows list of Dataloader corresponding to light sources and data formats currently implemented within piva, along with the most recent dates when the Dataloader was tested.

Dataloader	Facility	Date	File formats	Comments
Pickle	–	Oct 2023	pickle files	files saved with piva
SIS	SLS	Oct 2023	Scienta SES files (.zip, .ibw, .pxt) HDF file (.h5)	–
ADRESS	SLS	Aug 2023	HDF file (.h5)	–
Bloch	MAX IV	Oct 2023	Scienta SES files (.zip, .ibw, .pxt)	–
MERLIN	ALS	Apr 2023	Scienta SES files (.zip, .ibw, .pxt) HDF file (.h5)	–
I05	Diamond	Nov 2022	NeXus (.nxs)	–
URANOS	Solaris	Apr 2023	Scienta SES files (.zip, .ibw, .pxt)	–
CASSIOPEE	Soleil	Oct 2023	Scienta SES files (.zip, .ibw, .pxt) Text files (.txt)	–

Writing a custom Dataloader

In order to implement custom Dataloader one needs to:

1. write a loading script,

2. make it available for piva.

Description below shows step by step how to achieve both.

1. A piva dataloader is nothing else than a subclass of Dataloader that takes care of three things:

   • Reading an ARPES data file.

   • Extracting all relevant data and metadata from loaded file.

   • Creating Dataset object and filling it with so extracted data.

   The Dataloader class offers an interface for that. In order to write a custom dataloader, let’s call it CustomDataloader, one needs to start by creating a subclass of Dataloader:

   from piva.data_loaders import Dataloader
   
   class CustomDataloader(Dataloader):
       name = 'Custom1'
   
       def __init__(self):
           super(CustomDataloader1, self).__init__()
   

   As an abstract base class, Dataloader requires a load_data() method. Any custom subclass must therefore implement this method and assign a Dataset object to the self.ds attribute:

   def load_data(self, filename, metadata=False):
       # <Your code here>
       # 1) Read in data from *filename*
       # 2) Extract necessary (meta)data
       # 3) Put it into an argparse.Namespace and return it
       return self.ds
   

   Note

   Both:

   • filename, str, absolut path to the file

   • metadata, bool, determine whether method should load entire dataset or just its metadata

   have to be arguments of the load_data() method. See load_data() for more details.

   See also

   See load_custom_data_loaders() for more details.

2. Making CustomDataloader visible for piva is accomplished through DataloaderImporter object. Its proper configuration requires following conditions:

   • The module (python file) in which DataloaderImporter is defined must be called piva_dataloader_importer.py.

   • Path to the piva_dataloader_importer.py needs to be added to the $PYTHONPATH of your virtual environment. A simple guide on how to do it on any operating system can be found here.

   At the beginning of the session, piva will search for the DataloaderImporter class and execute whatever code it contains. Correctly loaded CustomDataloader should appear in DataBrowser’s drop-down menu:

Note

The advantage of this approach is that the piva_dataloader_importer.py module and CustomDataloaders can be stored in any directory, without modifying the piva package. Moreover, once correctly implemented, the loader will be automatically imported at the beginning of each piva session and available to the user immediately.

Note

It is highly recommended to use the prepared templates for both DataloaderImporter and CustomDataloader, which can be found here.

Example

Users are free to implement custom data loaders according to their preferences. However, they may find the provided template here helpful. The template offers a basic implementation of the DataloaderImporter and CustomDataloader classes.

Additionally, the template can be used to test the configuration with an example spectrum file, which contains a simulated 2D spectrum:

Contributing

We encourage everyone to share their tested, self-written data loaders with the community by adding them to piva’s source code. This can be done ideally directly through GitHub or alternatively by contacting the development team:

Wojtek Pudelko and Kevin Kramer.