dataframe

DataFrame class

class dataframe.DataFrame

DataFrame class. The raccoon DataFrame implements a simplified version of the pandas DataFrame with the key objective difference that the raccoon DataFrame is meant for use cases where the size of the DataFrame rows is expanding frequently. This is known to be slow with Pandas due to the use of numpy as the underlying data structure. Raccoon uses BList as the underlying data structure which is quick to expand and grow the size. The DataFrame can be designated as sort, in which case the rows will be sort by index on construction, and then any addition of a new row will insert it into the DataFrame so that the index remains sort.

__init__(data=None, columns=None, index=None, index_name="index", use_blist=False, sort=None)
Parameters:data – (optional) dictionary of lists. The keys of the dictionary will be used for the column names and

the lists will be used for the column data. :param columns: (optional) list of column names that will define the order :param index: (optional) list of index values. If None then the index will be integers starting with zero :param index_name: (optional) name for the index. Default is “index” :param use_blist: if True then use blist() as the underlying data structure, if False use standard list() :param sort: if True then DataFrame will keep the index sort. If True all index values must be of same type

__repr__()
__str__()
_make_table(index=True, **kwargs)
show(index=True, **kwargs)

Print the contents of the DataFrame. This method uses the tabulate function from the tabulate package. Use the kwargs to pass along any arguments to the tabulate function.

Parameters:
  • index – If True then include the indexes as a column in the output, if False ignore the index
  • kwargs – Parameters to pass along to the tabulate function
Returns:

output of the tabulate function

_sort_columns(columns_list)

Given a list of column names will sort the DataFrame columns to match the given order

Parameters:columns_list – list of column names. Must include all column names
Returns:nothing
_pad_data(max_len=None)

Pad the data in DataFrame with [None} to ensure that all columns have the same length.

Parameters:max_len – If provided will extend all columns to this length, if not then will use the longest column
Returns:nothing
__len__()
data()
columns()
columns(columns_list)
index()

Return a view of the index as a list. Because this is a view any change to the return list from this method will corrupt the DataFrame.

Returns:list
index(index_list)
index_name()
index_name(name)
blist()
sort()
sort(boolean)
select_index(compare, result="boolean")

Finds the elements in the index that match the compare parameter and returns either a list of the values that match, of a boolean list the length of the index with True to each index that matches. If the indexes are tuples then the compare is a tuple where None in any field of the tuple will be treated as “*” and match all values.

Parameters:
  • compare – value to compare as a singleton or tuple
  • result – ‘boolean’ = returns a list of booleans, ‘value’ = returns a list of index values that match
Returns:

list of booleans or values

get(indexes=None, columns=None, as_list=False, as_dict=False)

Given indexes and columns will return a sub-set of the DataFrame. This method will direct to the below methods based on what types are passed in for the indexes and columns. The type of the return is determined by the types of the parameters.

Parameters:
  • indexes – index value, list of index values, or a list of booleans. If None then all indexes are used
  • columns – column name or list of column names. If None then all columns are used
  • as_list – if True then return the values as a list, if False return a DataFrame. This is only used if the get is for a single column
  • as_dict – if True then return the values as a dictionary, if False return a DataFrame. This is only used if the get is for a single row
Returns:

either DataFrame, list, dict or single value. The return is a shallow copy

get_cell(index, column)

For a single index and column value return the value of the cell

Parameters:
  • index – index value
  • column – column name
Returns:

value

get_rows(indexes, column, as_list=False)

For a list of indexes and a single column name return the values of the indexes in that column.

Parameters:
  • indexes – either a list of index values or a list of booleans with same length as all indexes
  • column – single column name
  • as_list – if True return a list, if False return DataFrame
Returns:

DataFrame is as_list if False, a list if as_list is True

get_columns(index, columns=None, as_dict=False)

For a single index and list of column names return a DataFrame of the values in that index as either a dict or a DataFrame

Parameters:
  • index – single index value
  • columns – list of column names
  • as_dict – if True then return the result as a dictionary
Returns:

DataFrame or dictionary

get_entire_column(column, as_list=False)

Shortcut method to retrieve a single column all rows. Since this is a common use case this method will be faster than the more general method.

Parameters:
  • column – single column name
  • as_list – if True return a list, if False return DataFrame
Returns:

DataFrame is as_list if False, a list if as_list is True

get_matrix(indexes, columns)

For a list of indexes and list of columns return a DataFrame of the values.

Parameters:
  • indexes – either a list of index values or a list of booleans with same length as all indexes
  • columns – list of column names
Returns:

DataFrame

get_location(location, columns=None, as_dict=False, index=True)

For an index location and list of columns return a DataFrame of the values. This is optimized for speed because it does not need to lookup the index location with a search. Also can accept relative indexing from the end of the DataFrame in standard python notation [-3, -2, -1]

Parameters:
  • location – index location in standard python form of positive or negative number
  • columns – list of columns, or None to include all columns
  • as_dict – if True then return a dictionary
  • index – if True then include the index in the dictionary if as_dict=True
Returns:

DataFrame or dictionary

get_locations(locations, columns=None, **kwargs)

For list of locations and list of columns return a DataFrame of the values.

Parameters:
  • locations – list of index locations
  • columns – list of column names
  • kwargs – will pass along these parameters to the get() method
Returns:

DataFrame

get_slice(start_index=None, stop_index=None, columns=None, as_dict=False)

For sorted DataFrames will return either a DataFrame or dict of all of the rows where the index is greater than or equal to the start_index if provided and less than or equal to the stop_index if provided. If either the start or stop index is None then will include from the first or last element, similar to standard python slide of [:5] or [:5]. Both end points are considered inclusive.

Parameters:
  • start_index – lowest index value to include, or None to start from the first row
  • stop_index – highest index value to include, or None to end at the last row
  • columns – list of column names to include, or None for all columns
  • as_dict – if True then return a tuple of (list of index, dict of column names: list data values)
Returns:

DataFrame or tuple

_insert_row(i, index)

Insert a new row in the DataFrame.

Parameters:
  • i – index location to insert
  • index – index value to insert into the index list
Returns:

nothing

_insert_missing_rows(indexes)

Given a list of indexes, find all the indexes that are not currently in the DataFrame and make a new row for that index, inserting into the index. This requires the DataFrame to be sort=True

Parameters:indexes – list of indexes
Returns:nothing
_add_row(index)

Add a new row to the DataFrame

Parameters:index – index of the new row
Returns:nothing
_add_missing_rows(indexes)

Given a list of indexes, find all the indexes that are not currently in the DataFrame and make a new row for that index by appending to the DataFrame. This does not maintain sort order for the index.

Parameters:indexes – list of indexes
Returns:nothing
_add_column(column)

Add a new column to the DataFrame

Parameters:column – column name
Returns:nothing
set(indexes=None, columns=None, values=None)

Given indexes and columns will set a sub-set of the DataFrame to the values provided. This method will direct to the below methods based on what types are passed in for the indexes and columns. If the indexes or columns contains values not in the DataFrame then new rows or columns will be added.

Parameters:
  • indexes – indexes value, list of indexes values, or a list of booleans. If None then all indexes are used
  • columns – columns name, if None then all columns are used. Currently can only handle a single column or

all columns :param values: value or list of values to set (index, column) to. If setting just a single row, then must be adict where the keys are the column names. If a list then must be the same length as the indexes parameter, ifindexes=None, then must be the same and length of DataFrame :return: nothing

set_cell(index, column, value)

Sets the value of a single cell. If the index and/or column is not in the current index/columns then a new index and/or column will be created.

Parameters:
  • index – index value
  • column – column name
  • value – value to set
Returns:

nothing

set_row(index, values)

Sets the values of the columns in a single row.

Parameters:
  • index – index value
  • values – dict with the keys as the column names and the values what to set that column to
Returns:

nothing

set_column(index=None, column=None, values=None)

Set a column to a single value or list of values. If any of the index values are not in the current indexes then a new row will be created.

Parameters:index – list of index values or list of booleans. If a list of booleans then the list must be the same

length as the DataFrame :param column: column name :param values: either a single value or a list. The list must be the same length as the index list if the indexlist is values, or the length of the True values in the index list if the index list is booleans :return: nothing

set_location(location, values, missing_to_none=False)

Sets the column values, as given by the keys of the values dict, for the row at location to the values of the values dict. If missing_to_none is False then columns not in the values dict will be left unchanged, if it is True then they are set to None. This method does not add new columns and raises an error.

Parameters:
  • location – location
  • values – dict of column names as keys and the value as the value to set the row for that column to
  • missing_to_none – if True set any column missing in the values to None, otherwise leave unchanged
Returns:

nothing

set_locations(locations, column, values)

For a list of locations and a column set the values.

Parameters:
  • locations – list of index locations
  • column – column name
  • values – list of values or a single value
Returns:

nothing

append_row(index, values, new_cols=True)

Appends a row of values to the end of the data. If there are new columns in the values and new_cols is True they will be added. Be very careful with this function as for sort DataFrames it will not enforce sort order. Use this only for speed when needed, be careful.

Parameters:
  • index – value of the index
  • values – dictionary of values
  • new_cols – if True add new columns in values, if False ignore
Returns:

nothing

append_rows(indexes, values, new_cols=True)

Appends rows of values to the end of the data. If there are new columns in the values and new_cols is True they will be added. Be very careful with this function as for sort DataFrames it will not enforce sort order. Use this only for speed when needed, be careful.

Parameters:
  • indexes – list of indexes
  • values – dictionary of values where the key is the column name and the value is a list
  • new_cols – if True add new columns in values, if False ignore
Returns:

nothing

_slice_index(slicer)
__getitem__(index)

Convenience wrapper around the get() method for using df[] Usage... df[‘a’] – get column df[[‘a’,’b’,c’]] – get columns df[5, ‘b’] – get cell at index=5, column=’b’ df[[4, 5], ‘c’] – get indexes=[4, 5], column=’b’ df[[4, 5], [‘a’, ‘b’]] – get indexes=[4, 5], columns=[‘a’, ‘b’] Can also use a boolean list for anything. If the DataFrame is sort=True then the indexes slice values do not need to be in the index, will use greater than or equal to / less than equal to. For sort=False the provided slide values must be in the index.

Parameters:index – any of the parameters above
Returns:DataFrame of the subset slice
__setitem__(index, value)

Convenience wrapper around the set() method for using df[] = X Usage...

df[1, ‘a’] – set cell at index=1, column=a df[[0, 3], ‘b’] – set index=[0, 3], column=b df[1:2, ‘b’] – set index slice 1:2, column=b

Parameters:
  • index – any of the parameter examples above
  • value – single value or list of values
Returns:

nothing

to_list()

For a single column DataFrame returns a list of the values. Raises error if more then one column.

Returns:list
to_dict(index=True, ordered=False)

Returns a dict where the keys are the column names and the values are lists of the values for that column.

Parameters:
  • index – If True then include the index in the dict with the index_name as the key
  • ordered – If True then return an OrderedDict() to preserve the order of the columns in the DataFrame
Returns:

dict or OrderedDict()

to_json()

Returns a JSON of the entire DataFrame that can be reconstructed back with raccoon.from_json(input). Any object that cannot be serialized will be replaced with the representation of the object using repr(). In that instance the DataFrame will have a string representation in place of the object and will not reconstruct exactly.

Returns:json string
rename_columns(rename_dict)

Renames the columns

Parameters:rename_dict – dict where the keys are the current column names and the values are the new names
Returns:nothing
head(rows)

Return a DataFrame of the first N rows

Parameters:rows – number of rows
Returns:DataFrame
tail(rows)

Return a DataFrame of the last N rows

Parameters:rows – number of rows
Returns:DataFrame
delete_rows(indexes)

Delete rows from the DataFrame

Parameters:indexes – either a list of values or list of booleans for the rows to delete
Returns:nothing
delete_all_rows()

Deletes the contents of all rows in the DataFrame. This function is faster than delete_rows() to remove all information, and at the same time it keeps the container lists for the columns and index so if there is another object that references this DataFrame, like a ViewSeries, the reference remains in tact.

Returns:nothing
delete_columns(columns)

Delete columns from the DataFrame

Parameters:columns – list of columns to delete
Returns:nothing
sort_index()

Sort the DataFrame by the index. The sort modifies the DataFrame inplace

Returns:nothing
sort_columns(column, key=None, reverse=False)

Sort the DataFrame by one of the columns. The sort modifies the DataFrame inplace. The key and reverse parameters have the same meaning as for the built-in sort() function.

Parameters:
  • column – column name to use for the sort
  • key – if not None then a function of one argument that is used to extract a comparison key from each list element
  • reverse – if True then the list elements are sort as if each comparison were reversed.
Returns:

nothing

_validate_index(indexes)
_validate_columns(columns)
_validate_data()
validate_integrity()

Validate the integrity of the DataFrame. This checks that the indexes, column names and internal data are not corrupted. Will raise an error if there is a problem.

Returns:nothing
append(data_frame)

Append another DataFrame to this DataFrame. If the new data_frame has columns that are not in the current DataFrame then new columns will be created. All of the indexes in the data_frame must be different from the current indexes or will raise an error.

Parameters:data_frame – DataFrame to append
Returns:nothing
equality(column, indexes=None, value=None)

Math helper method. Given a column and optional indexes will return a list of booleans on the equality of the value for that index in the DataFrame to the value parameter.

Parameters:
  • column – column name to compare
  • indexes – list of index values or list of booleans. If a list of booleans then the list must be the same

length as the DataFrame :param value: value to compare :return: list of booleans

_get_lists(left_column, right_column, indexes)
add(left_column, right_column, indexes=None)

Math helper method that adds element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:
  • left_column – first column name
  • right_column – second column name
  • indexes – list of index values or list of booleans. If a list of booleans then the list must be the same

length as the DataFrame :return: list

subtract(left_column, right_column, indexes=None)

Math helper method that subtracts element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:
  • left_column – first column name
  • right_column – name of column to subtract from the left_column
  • indexes – list of index values or list of booleans. If a list of booleans then the list must be the same

length as the DataFrame :return: list

multiply(left_column, right_column, indexes=None)

Math helper method that multiplies element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:
  • left_column – first column name
  • right_column – second column name
  • indexes – list of index values or list of booleans. If a list of booleans then the list must be the same

length as the DataFrame :return: list

divide(left_column, right_column, indexes=None)

Math helper method that divides element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:
  • left_column – column name of dividend
  • right_column – column name of divisor
  • indexes – list of index values or list of booleans. If a list of booleans then the list must be the same

length as the DataFrame :return: list

isin(column, compare_list)

Returns a boolean list where each elements is whether that element in the column is in the compare_list.

Parameters:
  • column – single column name, does not work for multiple columns
  • compare_list – list of items to compare to
Returns:

list of booleans

iterrows(index=True)

Iterates over DataFrame rows as dictionary of the values. The index will be included.

Parameters:index – if True include the index in the results
Returns:dictionary
itertuples(index=True, name="Raccoon")

Iterates over DataFrame rows as tuple of the values.

Parameters:
  • index – if True then include the index
  • name – name of the namedtuple
Returns:

namedtuple

reset_index(drop=False)

Resets the index of the DataFrame to simple integer list and the index name to ‘index’. If drop is True then the existing index is dropped, if drop is False then the current index is made a column in the DataFrame with the index name the name of the column. If the index is a tuple multi-index then each element of the tuple is converted into a separate column. If the index name was ‘index’ then the column name will be ‘index_0’ to not conflict on print().

Parameters:drop – if True then the current index is dropped, if False then index converted to columns
Returns:nothing
from_json(json_string)

Creates and return a DataFrame from a JSON of the type created by to_json

Parameters:json_string – JSON
Returns:DataFrame