semlog_mongo API

semlog_mongo.mongo

In mongo is a class that manage for multi collectin queries. Can be initilized by an instance of class WebsiteData. Or mannually.

class mongo.MongoDB(database_collection_list, ip='127.0.0.1', port=27017, config_path=None)

Class for multi-collection query.

apply_similar_filtering(flag_class_apply_filtering, class_id_list, similar_dict)

Apply two stage similarity filtering.

Parameters

  • flag_class_apply_filtering – Whether apply class filtering or not.

  • class_id_list – A list of classes.

  • similar_dict – A dict contains parameters for filtering.

Function for searching MongoDB with entity conditions.

Parameters

  • object_id_list – a list of object_ids

  • class_id_list – a list of classes

  • object_pattern ('id'|'class') – Search with object ids or classes

  • object_logic ('and'|'or') – Search with which logic

  • image_type_list (['Color',"Depth","Mask","Normal"]) – type of images

  • flag_ignore_similar_image (True|False) –

  • flag_class_ignore_similar_image (True|False) –

Returns

A pandas Dataframe contains all qualified images.

get_all_object_id()

Get all unique object id from all collections.

get_object_rgb_dict(id_list, search_type)

Get a dict contains all class,object id and their colors.

Parameters

  • id_list – A list of classes or object ids

  • search_type – ‘class’ or ‘id’

Returns

A dictionary contains elements from id_list and their colors

class mongo.MongoDB(database_collection_list, ip='127.0.0.1', port=27017, config_path=None)

Class for multi-collection query.

apply_similar_filtering(flag_class_apply_filtering, class_id_list, similar_dict)

Apply two stage similarity filtering.

Parameters

  • flag_class_apply_filtering – Whether apply class filtering or not.

  • class_id_list – A list of classes.

  • similar_dict – A dict contains parameters for filtering.

entity_search(object_id_list=None, class_id_list=None, object_pattern='class', object_logic='or', image_type_list=None, flag_ignore_similar_image=False, flag_class_ignore_similar_image=False)

Function for searching MongoDB with entity conditions.

Parameters

  • object_id_list – a list of object_ids

  • class_id_list – a list of classes

  • object_pattern ('id'|'class') – Search with object ids or classes

  • object_logic ('and'|'or') – Search with which logic

  • image_type_list (['Color',"Depth","Mask","Normal"]) – type of images

  • flag_ignore_similar_image (True|False) –

  • flag_class_ignore_similar_image (True|False) –

Returns

A pandas Dataframe contains all qualified images.

get_all_object_id()

Get all unique object id from all collections.

get_object_rgb_dict(id_list, search_type)

Get a dict contains all class,object id and their colors.

Parameters

  • id_list – A list of classes or object ids

  • search_type – ‘class’ or ‘id’

Returns

A dictionary contains elements from id_list and their colors

semlog_mongo.utils

utils stores functions for one collection queries. They are the smallest components between Python and MongoDB. Many useful little functions are created here.

utils.calculate_per_image_similar(client, csv_saving_path=None)

Calculate the average parameters of similarity.

Parameters

  • client – A pymongo.MongoClient() instance.

  • csv_saving_path – Directory to save the result.

utils.check_and_update_similar(client, linear_distance_tolerance, angular_distance_tolerance)

Update duplicate flage in the database.

Parameters

  • client – A MongoClient instance

  • linear_distance_tolerance – Threshold for total linear distance.

  • angular_distance_tolerance – Threshold for total angular distance.

utils.check_and_update_similar_per_class(client, class_list, num_pixels_tolerance=150, linear_distance_tolerance=0.05, angular_distance_tolerance=0.005, flag_ignore_similar_image=False)

Main function for filtering duplicate objects per class.

Parameters

  • client – A MongoClient instance.

  • class_list – A list of classes.

  • num_pixels_tolerance – Threshold for num_pixels.

  • linear_distance_tolerance – Threshold for linear distance.

  • Angular_distance_tolerance – Threshold for angular distance.

  • flag_ignore_similar_image – Whether to ignore similar images or not.

utils.download_all_object_name(client, saving_path='all_objects.csv')

Get all object infomation from meta collection.

Parameters

  • client – A MongoClient instance.

  • saving_path – Path to save the csv file.

utils.download_images(ip, root_folder_path, root_folder_name, df, config_path=None)

Donwload all images via a DataFrame.

Parameters

  • ip – The ip address to the MongoDB server.

  • root_folder_path – Path to save the root folder.

  • root_folder_name – Folder name of saving all images.

  • df – A result DataFrame from a search method.

  • config_path – Path of config file for pwd and user name.

utils.download_one(download_db, image, abs_path='', header='')

Multiprocessing version of download.

Parameters

  • download_db – A GridFS instance of the target collection. Example: download_db = gridfs.GridFSBucket(MongoClient(ip)[database], collection)

  • image – A list contains file_id and type.

  • abs_path – Path of root folder.

  • header – Name of root folder.

Search with event sentences.

Parameters

  • ip – Target ip address.

  • view_list – a lsit of event sentences contains db-collection-timestamp-camera_view_name

  • config_path – File records account information.

Returns

A df contains qualified results.

utils.get_all_object_from_meta(client)

Get all object ids from the meta collection.

Parameters

client – A MongoClient with meta collection.

Returns

A list of object ids.

utils.get_class_by_object_id(client, object_id)

Get class name from a given object id.

Parameters

  • client – A MongoClient instance

  • object_id – A string to search for its class.

utils.get_object_by_class(client, class_list)

Get all objects from a given class list

Parameters

  • client – A MongoClient instance.

  • class_list – List of class names.

Returns

A list of object ids.

utils.get_object_list_by_class_list(client, class_list)

Get all objects from a given class list.

Parameters

  • client – A MongoClient instance.

  • class_list – A list of classes.

Returns

A dict with keys as classes and values as objects.

utils.get_object_rgb(client, object_id)

Retrieve the rgb of an object.

Parameters

  • object_id (str) – Id for a specific object.

  • collection (str) – Name of the collection for access meta data.

utils.get_scans_by_class(meta_client, class_name=None, image_type_list=None)

Get scan images by class name.

Parameters

  • meta_client – A MongoClient instance.

  • class_name – Name of the class in the meta collection.

Returns

A list of images.

utils.load_mongo_account(config_path)

Load account info from a local file.

Search for scan images.

Parameters

  • ip – The ip address to the MongoDB server.

  • db_collection – A pair of database and collection.

  • scan_class_list – A list contains class names.

  • image_type_list – A list conatians image types.

  • config_path – Path to config file for user name and pwd.

Returns

A result DataFrame.

utils.search(client, object_id_list=None, image_type_list=None, percentage=0.0001, image_limit=None, flag_ignore_similar_image=False, flag_ignore_similar_object=False)

Input params to search for specific images.

Parameters

  • object_id_list – A list of object ids.

  • image_type_list – type for images.

  • percentage – least percentage of objects in images.

  • image_limit – number of images to be download.

  • flag_ignore_similar_image – Whether to ignore similar images or not.

  • flag_ingore_similar_object – Whether to ignore similar objects or not.

Returns

Result list from the MongoDB.

utils.search_class_by_and(client, class_id_list=None, image_type_list=None, image_limit=None, flag_ignore_similar_image=False)

Search with and logic and class list.

Parameters

  • client – The MongoClient instance.

  • class_id_list – A list of classes.

  • image_type_list – A list of image types.

  • flag_ignore_similar_image – Whether ignore similar image or not.

Returns

A list of qualified results.

utils.search_object_by_and(client, object_id_list=None, image_type_list=None, image_limit=None, flag_ignore_similar_image=False)

Search with and logic and object id list.

Parameters

  • client – The MongoClient instance.

  • object_id_list – A list of classes.

  • image_type_list – A list of image types.

  • flag_ignore_similar_image – Whether ignore similar image or not.

Returns

A list of qualified results.

utils.search_single_image_by_view(client, timestamp, view_id)

Search with the give camera name and timestamp.

Parameters

  • client – A MongoClient instance.

  • timestamp – Search for the first image closest to this timestamp.

  • view_id – Class name of camera_views

Returns

A list of qualified results.

semlog_mongo.analysis

analysis is used for analyzing data from MongoDB database.

analysis.get_image_information(client)

This function is used for data analysis. Retrieve image information (num_entities, average linear/angular distance and timestamp).

Parameters

client – A MongoClient to search for.

Returns

A list of dicts contains average information.