Event processing

A collection of utility functions for loading, cleaning, normalizing, and combining events. There are also a smattering of other helper function for selecting specific types of events. In general, the following steps must be taken to go from raw (on-disk) events to events that can be analyzed:

1. Load: Load events from disk into memory
2. Clean: Perform standard sets of cleaning operations
3. Normalize: Modify fields and values so that events from different experiment can be easily combined

ramutils.events.add_field(events, field_name, default_val, dtype)

Add field to the recarray

Notes

Converting to a dataframe, adding the field, and reconverting to a recarray because the rec_append_fields function in numpy doesn’t seem to work

ramutils.events.add_list_phase_info(events)

Adds a list_phase field to an event structure that says which phase of the list (ENCODING, DISTRACT, RETRIEVAL,…) that event is part of.

Parameters:events (np.recarray) – All event or task events

ramutils.events.build_dtype_list(dtypes)

Given a numpy.dtype object, return a list of tuples in the form (field_name, field_type_string)

ramutils.events.calculate_repetition_ratio(recall_events)

Determine the repetition ratio for a given list based on the recalled events for that list

ramutils.events.clean_events(events, start_time=None, end_time=None, duration=None, pre=None, post=None, return_stim_events=False, all_events=False)

Peform basic cleaning operations on events such as removing incomplete sessions, negative offset events, and incomplete lists. For FR events, baseline events needs to be found. Events are then normalized so that cross-experiment events can be merged.

Parameters:

• events (np.recarray) – Raw events
• start_time (int) –
• end_time (int) –
• duration (int) –
• pre (int) –
• post (int) –
• return_stim_events (bool) – Indicator for if stim parameters should be returned in addition to the cleaned events
• all_events (bool) – Indicates if the data to be cleaned is the all_event.json file. These require a different set of cleaning procedures

Returns:

Cleaned set of events

Return type:

np.recarray

Notes

This function should be called on an experiment by experiment basis and should not be used to clean cross-experiment datasets

ramutils.events.coerce_study_pair_to_word_event(events)

Update STUDY_PAIR events to be WORD events. These are the same event type, but PAL calls them STUDY_PAIR and FR/catFR call them WORD. In the future, it may make more sense to make an update to event creation instead of coercing the event types here.

ramutils.events.combine_retrieval_events(events)

Combine baseline retrieval and actual retrieval events into a single event type.

ramutils.events.concatenate_events_across_experiments(event_list, pal=False, stim=False, cat=False)

Concatenate events across different experiment types. To make session numbers unique, 100 is added to the second set of events in event_list, 200 to the next set of events, and so on.

Parameters:

• event_list (iterable) – An iterable containing events to be concatenated
• pal (Bool) – Indicator for if PAL sessions are included in event_list. This will alter which columns are kept for merging events
• stim (Bool) – Indicator for if event_list contains stim sessions. If True, then stim_params field will be kept

Returns:

The combined set of events

Return type:

np.recarray

ramutils.events.concatenate_events_for_single_experiment(event_list)

Combine events that are part of the same experiment

Parameters:event_list – Returns:The flattened set of events Return type:np.recarray

ramutils.events.correct_fr2_stim_item_identification(stim_param_df)

Update the stim_item and post_stim_item masks for FR2 stim experiments

The FR2 experiment is a special bird in that stimulation occurs across two items at a time, and therefore only a single STIM_ON event is recorded. This causes the stim item identification algorithm to miss those second items and therefore they must be corrected separately

stim_param_df: pd.DataFrame

Table containing the fully processed encoding events

Returns:stim_param_df – DataFrame with corrected is_stim_item and is_post_stim_item fields Return type:pd.DataFrame

ramutils.events.dataframe_to_recarray(dataframe, dtypes)

Convert from dataframe to recarray maintaining the original datatypes

ramutils.events.extract_event_metadata(events)

Extract the subject, experiment(s), and session(s) associated with an event structure

ramutils.events.extract_experiment_from_events(events)

Given a set of events, return a list of unique experiments contained within

ramutils.events.extract_lists(events)

Return a list of lists contained within the events structure

ramutils.events.extract_sample_rate_from_eeg(events)

Extract the samplerate used for the given set of events by loading EEG

ramutils.events.extract_sessions(events)

Return a list of sessions contained within the events structure

ramutils.events.extract_stim_information(all_events, task_events)

Identify stim items, post stim items, and stimulation parameters

Parameters:

• all_events (np.recarray) – All events with stim_params field
• task_events (np.recarray) – Task events used for classifier training/evaluation

Returns:

• is_stim_item (list) – Boolean array matching the length of task events indicating if a word was stimulated
• is_post_stim_item (list) – Boolean array matching the length of task_events indicating if a word occured after a stimulated word
• stim_df (pd.DataFrame) – Stim parameters used for each stimulation event

Notes

This is a rather convoluted set of logic. The goal is to match all word encoding events with their associated STIM_ON events, which occur as separate entries in the json event structures.

ramutils.events.extract_subject(events, add_localization=False)

Extract subject identifier from events

ramutils.events.find_free_time_periods(times, duration, pre, post, start=None, end=None)

Given a list of event times, find epochs between them when nothing is happening.

Parameters:

• times (list where elements are lists) – An iterable of 1-d numpy arrays, each of which is a list that indicates the starting times of all vocalization events. We do not want to include these as candidate time periods
• duration (int) – The length of the desired empty epochs
• pre (int) – the time before each event to exclude
• post (int) – The time after each event to exclude
• start (array_like) – List of a recall period start times
• end (array_like) – List of recall period end times

Returns:

epoch_array

Return type:

np.ndarray

ramutils.events.find_subjects(experiment, rootdir='/')

Identify subjects who completed a given experiment

ramutils.events.get_all_retrieval_events_mask(events)

Create a boolean bask for any retrieval event

ramutils.events.get_baseline_retrieval_mask(events)

Create a boolean mask for baseline retrieval events

ramutils.events.get_encoding_mask(events)

Create encoding event mask

ramutils.events.get_fr_retrieval_events_mask(events)

Identify actual retrieval events for FR/catFR experiments

ramutils.events.get_math_events_mask(events)

Get a boolean array identifying math events

ramutils.events.get_nonstim_events_mask(events)

Get a mask of any non-stim WORD events

Notes

These events are what is used in post-hoc classifier evaluation

ramutils.events.get_pal_retrieval_events_mask(events)

Identify retrieval events for PAL experiments

ramutils.events.get_partition_masks(events)

Return a set of masks corresponding to the partitions present in the events

ramutils.events.get_recall_events_mask(events)

Create a boolean mask for any recall events

ramutils.events.get_required_columns(all_relevant=False, pal=False, stim=False, cat=False)

Return baseline mandatory columns based on experiment type

Keyword Arguments:  

• all_relevant (bool) – A subset that includes all fields that are subsequently used by any of the experiments
• pal (bool) – Fields specific to PAL experiments
• stim (bool) – Fields specific to stim experiments
• cat (bool) – Fields specific to categorical free recall experiments

ramutils.events.get_session_mask(events, session)

Return a mask for if an event belongs to the given session

ramutils.events.get_stim_list_mask(events)

Return boolean mask identifying stim lists

Notes

Not all items in a stim list will be stimulated. Stimulation will depend on the biomarker at the time of encoding

ramutils.events.get_stim_table_event_mask(events)

Return a mask of events to be included for building stim session summaries

ramutils.events.get_time_between_events(events)

Calculate the time between successive events

ramutils.events.get_vocalization_mask(events)

Create mask for vocalization events

ramutils.events.get_word_event_mask(events, encoding_only)

Get a mask identify word events. If encoding_only, then retrieval events will not be counted

ramutils.events.initialize_empty_event_reccarray()

Utility function for generating a recarray that looks normalized, but is empty.

ramutils.events.initialize_empty_stim_reccarray()

Generate empty recarray that mirrors fields in stim_params

ramutils.events.insert_baseline_retrieval_events(events, start_time, end_time, duration, pre, post, use_deprecated=False)

Match recall events to matching baseline periods of failure to recall. This is required for all free recall events, but is not necessary for PAL events, which have a natural baseline/comparison group. Baseline events all begin at least 1000 ms after a vocalization, and end at least 1000 ms before a vocalization. Each recall event is matched, wherever possible, to a valid baseline period from a different list within 3 seconds relative to the onset of the recall period.

Parameters:

• events (np.recarray) – The event structure in which to incorporate these baseline periods
• start_time (int) – The amount of time to skip at the beginning of the session (ms)
• end_time (int) – The amount of time within the recall period to consider (ms)
• duration (int) – The length of desired empty epochs
• pre (int) – The time before each event to exclude
• post (int) – The time after each event to exclude

Returns:

Events with REC_BASE event types inserted

Return type:

np.reccarray

ramutils.events.insert_baseline_retrieval_events_deprecated(events, start_time, end_time, duration, pre, post)

Match recall events to matching baseline periods of failure to recall. This is required for all free recall events, but is not necessary for PAL events, which have a natural baseline/comparison group. Baseline events all begin at least 1000 ms after a vocalization, and end at least 1000 ms before a vocalization. Each recall event is matched, wherever possible, to a valid baseline period from a different list within 3 seconds relative to the onset of the recall period.

Parameters:

• events (np.recarray) – The event structure in which to incorporate these baseline periods
• start_time (int) – The amount of time to skip at the beginning of the session (ms)
• end_time (int) – The amount of time within the recall period to consider (ms)
• duration (int) – The length of desired empty epochs
• pre (int) – The time before each event to exclude
• post (int) – The time after each event to exclude

Returns:

Events with REC_BASE event types inserted

Return type:

np.reccarray

ramutils.events.insert_baseline_retrieval_events_logan(events, duration, pre, post)

Match recall events to matching baseline periods of failure to recall. This is required for all free recall events, but is not necessary for PAL events, which have a natural baseline/comparison group. Baseline events all begin at least 1000 ms after a vocalization, and end at least 1000 ms before a vocalization. Each recall event is matched, wherever possible, to a valid baseline period from a different list within 3 seconds relative to the onset of the recall period.

Parameters:

• events (np.recarray) – The event structure in which to incorporate these baseline periods
• duration (int) – The length of desired empty epochs
• pre (int) – The time before each event to exclude
• post (int) – The time after each event to exclude

Returns:

Events with REC_BASE event types inserted

Return type:

np.reccarray

ramutils.events.load_events(subject, experiment, file_type='all_events', sessions=None, rootdir='/')

Load events for a specific subject and experiment. If no events are found, an empty recarray with the correct datatypes are returned

Parameters:

• subject (str) –
• experiment (str) –
• file_type (str) – The name of the event file to load, i.e. all_events, task_events, math_events, ps4_events. Default is ‘all_events’
• sessions (iterable or None) –
• rootdir (str) –

Returns:

A numpy recarray containing all events for the requested subject, experiment, and session(s)

Return type:

np.rec.array

ramutils.events.lookup_sample_rate(subject, experiment, session, rootdir='/')

Identify the sample rate used for a session

ramutils.events.normalize_pal_events(events)

Perform any normalization to PAL event so make the homogeneous enough so that it is trivial to combine with other experiment events.

ramutils.events.partition_events(events)

Split a given set of events into partitions by experiment class ( FR/PAL) and encoding/retrieval

Parameters:events (np.recarray) – Set of events to partition Returns:A list containing all identified partitions to the data Return type:list

ramutils.events.remove_bad_events(events)

Remove events whose offset values would result in trying to read data that is out of bounds in the EEG file. Currently, this is done automatically in PTSA when you load the EEG, but to avoid having to catch updated events when reading the EEG, it should be done ahead of time.

ramutils.events.remove_incomplete_lists(events)

Remove incomplete lists for every session in the given events. Note, there are two ways that this is done in the reporting code, so it is an outstanding item to determine which method is better

ramutils.events.remove_intrusions(events)

Select all encoding events that were part of the encoding period or were non-intrusion retrieval events.

ramutils.events.remove_negative_offsets(events)

Remove events with a negative eegoffset

ramutils.events.remove_nonresponses(events)

Selects only events that were listed as recalled or not recalled

ramutils.events.remove_practice_lists(events)

Remove practice lists from the set of events

ramutils.events.remove_session_number_offsets(experiment, sessions)

Given a list of sessions to include, undo the offsets for catFR and PAL so the sessions can be looked up correctly in the r1.json file

ramutils.events.remove_voice_detection(events)

Remove events

ramutils.events.rename_correct_to_recalled(events)

Normalizes PAL “recall” event names to match those of FR experiments.

Parameters:events (np.recarray) – Returns:Events with a ‘recalled’ field added to mirror the ‘correct’ field Return type:np.recarray

ramutils.events.select_all_retrieval_events(events)

Select both baseline and actual retrieval events

ramutils.events.select_baseline_retrieval_events(events)

Select baseline retrieval events

ramutils.events.select_column_subset(events, all_relevant=False, pal=False, stim=False, cat=False)

Select only the necessary subset of the fields

Parameters:

events (np.recaarray) – The set of events to subset from

Keyword Arguments:  

• all_relevant (bool) – A subset that includes all fields that are subsequently used by any of the experiments
• pal (bool) – Fields specific to PAL experiments
• stim (bool) – Fields specific to stim experiments
• cat (bool) – Fields specific to categorical free recall experiments

ramutils.events.select_encoding_events(events)

Select only encoding events

ramutils.events.select_math_events(events)

Select math events from a set of events

ramutils.events.select_retrieval_events(events)

Select retrieval events. Uses the experiment field in the events to determine how selection should be done since selection differes for PAL and FR/catFR

Parameters:events (np.recarray) – Events to mask

ramutils.events.select_session_events(events, session)

Select events corresponding to a particular session

ramutils.events.select_stim_table_events(events)

Return the events needed to build stim session summaries

ramutils.events.select_vocalization_events(events)

Select all vocalization events

ramutils.events.select_word_events(events, encoding_only=True)

Filter out any non-word events

Parameters:

• events (np.recarray) –
• encoding_only (bool) – Flag for whether retrieval events should be included

ramutils.events.separate_stim_events(events, pal=False, stim=True, cat=False)

Separate stim params contained within events structure from the 1-D

events. The returned events and stim_params are both 1-dimensional

Parameters:

• pal –
• stim –
• cat –
• events (np.recarray) – Event structure

Returns:

• events (np.reccary) – 1D event structure with stim params removed
• stim_params (np.recarray) – 2D stim params strsucture

ramutils.events.subset_pal_events(events)

Only a subset of event types are needed for PAL experiments

ramutils.events.update_pal_retrieval_events(events)

Create surrogate responses for retrieval period based on PS4/PAL5 design doc. Surrogate responses are created by identifying trials without any responses. For these trials, a new response time is created based on a random draw from the set of response times from actual responses.

ramutils.events.update_recall_outcome_for_retrieval_events(events)

Manually override the recall outcomes for baseline retrieval and word retrieval events. All baseline retrieval events should be marked as not recalled and all word events in the recall period should be marked as recalled. This assumes that intrusions have already been removed from the given set of events. It exists merely to serve as an extra check on what should already be true in the raw events data.

Parameters:events (np.recarray) – Returns:Events containing updated recall outcomes for retrieval events Return type:np.recarray

ramutils.events.update_subject(events)

Ensure subject field is populated for all events

ramutils.events.validate_single_experiment(events)

Raises an error if more than one experiment is present in the events

ramutils.events.validate_single_session(events)

Raises an error if more than one session is present in the events