Summary
I’m working on a very large data upload (10TB / 10 million images) and am having trouble running a simple upload of 3 images on the Ubuntu 20.04 command line using process_csv and upload. I patched mapillary_tools and fixed the issues in process_csv and upload so that process_csv and upload now run without errors, but they still do not upload anything because the metadata isn’t being created properly for the images by process_csv. I can continue to debug and rewrite mapillary_tools, but I am worried that even if I get the metadata for the images to be written properly, the upload will not work for reasons out of my control.
Do you know of anyone who has successfully used mapillary_tools to process CSV files and upload images that I could talk to about how they did it? Or do you have any alternate scalable strategy for uploading 10TB (10 million images)?
Details
Steps I’ve taken so far:
- I installed mapillary_tools as in the readme, using python3 and pip3 as suggested. There were many python errors for attempting to index iterables (e.g. zips and maps are iterables and not subscriptable in python3 as they are in python2).
- I patched mapillary_tools and fixed the issues in process_csv and upload so that process_csv and upload now run on the command line without errors, but they just skip over the images and do not upload them.
- The specific issue I’m currently running into is that preform_upload() in uploader.py checks whether the corresponding file in /home/ubuntu/mapillary_uploader/test/images/.mapillary/ exists, but unfortunately this directory is never created despite process_csv seeming to have run successfully.
- My next step would be digging into exif_write.py to see why the metadata is not being created.
My patch of mapillary_tools is in Trying to fix process_csv and upload from mapillary_tools (abd71bcc) · Commits · safe7y / Mapillary Uploader · GitLab.
Output from running my modified process_csv, upload, and checking for the expected Mapillary logs:
[I] ubuntu@ip-172-31-85-54~/mapillary_uploader> mapillary_tools process_csv --import_path “test/images”
–csv_path “/home/ubuntu/mapillary_uploader/test/mapillary.csv”
–filename_column 1
–timestamp_column 2
–latitude_column 3
–longitude_column 4
–advanced
**process_csv( {‘import_path’: ‘test/images’, ‘csv_path’: ‘/home/ubuntu/mapillary_uploader/test/mapillary.csv’, ‘filename_column’: 1, ‘timestamp_column’: 2, ‘latitude_column’: 3, ‘longitude_column’: 4, ‘heading_column’: None, ‘altitude_column’: None, ‘gps_week_column’: None, ‘time_format’: ‘%Y:%m:%d %H:%M:%S.%f’, ‘convert_gps_time’: False, ‘convert_utc_time’: False, ‘delimiter’: ‘,’, ‘header’: False, ‘meta_columns’: None, ‘meta_names’: None, ‘meta_types’: None, ‘verbose’: False, ‘keep_original’: False} )
Inserting csv data in image EXIF: 0it [00:00, ?it/s]**parse_csv_geotag_data( {‘csv_data’: [(‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003341-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003366-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003350-eval.jpg’), (‘2019:09:30 19:16:50.500000’, ‘2019:09:30 19:16:51.333333’, ‘2019:09:30 19:16:50.791667’), (‘40.8383’, ‘40.8385’, ‘40.8385’), (‘-115.779’, ‘-115.7788’, ‘-115.7788’)], ‘image_index’: 0, ‘column_indexes’: [0, 1, 2, 3, None, None, None], ‘convert_gps_time’: False, ‘convert_utc_time’: False, ‘time_format’: ‘%Y:%m:%d %H:%M:%S.%f’} )
[parse_csv_geotag_data]column_indexes: [0, 1, 2, 3, None, None, None]
[parse_csv_geotag_data]csv_data: [(‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003341-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003366-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003350-eval.jpg’), (‘2019:09:30 19:16:50.500000’, ‘2019:09:30 19:16:51.333333’, ‘2019:09:30 19:16:50.791667’), (‘40.8383’, ‘40.8385’, ‘40.8385’), (‘-115.779’, ‘-115.7788’, ‘-115.7788’)]
[parse_csv_geotag_data]image_index: 0
[process_csv] exif_edit.write(filename=filename)
**parse_csv_geotag_data( {‘csv_data’: [(‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003341-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003366-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003350-eval.jpg’), (‘2019:09:30 19:16:50.500000’, ‘2019:09:30 19:16:51.333333’, ‘2019:09:30 19:16:50.791667’), (‘40.8383’, ‘40.8385’, ‘40.8385’), (‘-115.779’, ‘-115.7788’, ‘-115.7788’)], ‘image_index’: 2, ‘column_indexes’: [0, 1, 2, 3, None, None, None], ‘convert_gps_time’: False, ‘convert_utc_time’: False, ‘time_format’: ‘%Y:%m:%d %H:%M:%S.%f’} )
[parse_csv_geotag_data]column_indexes: [0, 1, 2, 3, None, None, None]
[parse_csv_geotag_data]csv_data: [(‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003341-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003366-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003350-eval.jpg’), (‘2019:09:30 19:16:50.500000’, ‘2019:09:30 19:16:51.333333’, ‘2019:09:30 19:16:50.791667’), (‘40.8383’, ‘40.8385’, ‘40.8385’), (‘-115.779’, ‘-115.7788’, ‘-115.7788’)]
[parse_csv_geotag_data]image_index: 2
[process_csv] exif_edit.write(filename=filename)
**parse_csv_geotag_data( {‘csv_data’: [(‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003341-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003366-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003350-eval.jpg’), (‘2019:09:30 19:16:50.500000’, ‘2019:09:30 19:16:51.333333’, ‘2019:09:30 19:16:50.791667’), (‘40.8383’, ‘40.8385’, ‘40.8385’), (‘-115.779’, ‘-115.7788’, ‘-115.7788’)], ‘image_index’: 1, ‘column_indexes’: [0, 1, 2, 3, None, None, None], ‘convert_gps_time’: False, ‘convert_utc_time’: False, ‘time_format’: ‘%Y:%m:%d %H:%M:%S.%f’} )
[parse_csv_geotag_data]column_indexes: [0, 1, 2, 3, None, None, None]
[parse_csv_geotag_data]csv_data: [(‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003341-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003366-eval.jpg’, ‘/home/ubuntu/mapillary_uploader/test/images/190930_01_pf_C0005_0000003350-eval.jpg’), (‘2019:09:30 19:16:50.500000’, ‘2019:09:30 19:16:51.333333’, ‘2019:09:30 19:16:50.791667’), (‘40.8383’, ‘40.8385’, ‘40.8385’), (‘-115.779’, ‘-115.7788’, ‘-115.7788’)]
[parse_csv_geotag_data]image_index: 1
[process_csv] exif_edit.write(filename=filename)
[I] ubuntu@ip-172-31-85-54~/mapillary_uploader> mapillary_tools upload --import_path “/home/ubuntu/mapillary_uploader/test/images/”
[uploader]get_upload_file_list import_path /home/ubuntu/mapillary_uploader/test/images/
[uploader]get_upload_file_list os.walk(import_path): [(‘/home/ubuntu/mapillary_uploader/test/images/’, , [‘190930_01_pf_C0005_0000003341-eval.jpg’, ‘190930_01_pf_C0005_0000003350-eval.jpg’, ‘190930_01_pf_C0005_0000003366-eval.jpg’])]
[uploader]get_upload_file_list files:: [‘190930_01_pf_C0005_0000003341-eval.jpg’, ‘190930_01_pf_C0005_0000003350-eval.jpg’, ‘190930_01_pf_C0005_0000003366-eval.jpg’]
[uploader]get_upload_file_list examining: 190930_01_pf_C0005_0000003341-eval.jpg
[uploader]preform_upload process_success: /home/ubuntu/mapillary_uploader/test/images/.mapillary/logs/190930_01_pf_C0005_0000003341-eval/mapillary_image_description_success
[uploader]preform_upload not os.path.isfile(upload_success): False
[uploader]preform_upload os.path.isfile(process_success): False
[uploader]preform_upload not os.path.isfile(duplicate): False
[uploader]get_upload_file_list examining: 190930_01_pf_C0005_0000003350-eval.jpg
[uploader]preform_upload process_success: /home/ubuntu/mapillary_uploader/test/images/.mapillary/logs/190930_01_pf_C0005_0000003350-eval/mapillary_image_description_success
[uploader]preform_upload not os.path.isfile(upload_success): False
[uploader]preform_upload os.path.isfile(process_success): False
[uploader]preform_upload not os.path.isfile(duplicate): False
[uploader]get_upload_file_list examining: 190930_01_pf_C0005_0000003366-eval.jpg
[uploader]preform_upload process_success: /home/ubuntu/mapillary_uploader/test/images/.mapillary/logs/190930_01_pf_C0005_0000003366-eval/mapillary_image_description_success
[uploader]preform_upload not os.path.isfile(upload_success): False
[uploader]preform_upload os.path.isfile(process_success): False
[uploader]preform_upload not os.path.isfile(duplicate): False
[get_upload_file_list]
[upload]upload_file_list:
No images to upload.
Please check if all images contain the required Mapillary metadata. If not, you can use “mapillary_tools process” to add them
[I] ubuntu@ip-172-31-85-54~/.c/f/functions> ls /home/ubuntu/mapillary_uploader/test/images/.mapillary
ls: cannot access ‘/home/ubuntu/mapillary_uploader/test/images/.mapillary’: No such file or directory