Hello there!
Recently I’ve decided to participate in OpenEDS 2020 challenge. The organizers of the competition offer two tracks:
- Predicting future gaze locations based on the previously estimated gaze vectors
- Predicting future gaze locations based on the previously estimated gaze vectors while additionally leveraging spatio-temporal information encoded in sequence of previously recorded eye-images
Each of which comes with its own dataset available by request to download from Dropbox. So how could we manage downloading datasets from Dropbox?
Headless Dropbox
The first path that I chose was to install Dropbox application on my Ubuntu server. Following this I successfully installed headless Dropbox and passed authorization.
It was possible to start daemon with ~/.dropbox-dist/dropboxd command but after that the reasonable question appeared “How does one check the current status of syncing?”. To get such functionality it is needed to download python script from the same page I’ve mentioned earlier and use it as executable that controls Dropbox on a server. As for me, the proposed control method looks ridicoulous but this is not the subject of this post.
With this script you could run dropbox status and you’ll know that it’s Syncing…. If everything goes right after some time you’ll obtain your dataset. In my case something went wrong and Dropbox daemon sent me an error:
Unable to monitor entire Dropbox folder hierarchy. Please run “echo fs.inotify.max_user_watches=100000 | sudo tee -a /etc/sysctl.conf; sudo sysctl -p” and restart Dropbox to fix the problem.Yup, that’s look like “Just do what they are asking you to do!” but it is not that easy. We use the server as a shared resource that’s why everyone works inside his or her own Docker container. The fs.inotify.max_user_watches setting is applicable to the whole host so I couldn’t change it within a container. This is the moment when the server administrator came to the stage. He proposed two options:
— I (means he) could change this setting system wide OR you can use rclone
Rclone [Solution]
To use rclone and connect it to my Dropbox account I followed this tutorial. It is needed to run rclone config and it will guide you through the configuration process.
Everything was smooth until running rclone authorize dropbox . This command launches server on a localhost that says:
If your browser doesn’t open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXXXXXXXXXXXXXX
Log in and authorize rclone for access
Waiting for code…But I have no browser in my terminal *(if you’re interested in “Could**w3m terminal browser handle it?”, the answer is: “No it can’t”).*Unfortunately, my Docker container was set up so that it was impossible to follow the link using the IP address of the host machine. Only servers (means applications) started with the 0.0.0.0 address is visible using the IP.
Dear Reader, If you know how to properly set up Docker container to do so, please comment this out
Luckily, rclone provides a possibility to use config file obtained on the other machine. That’s why I installed rclone locally and did everything described above to obtain proper config. After that I was able to easily transfer contents of local /home/emil/.config/rclone/rclone.conf to remote/root/.config/rclone/rclone.conf (inside container).
And here it is the end of my two days journey for obtaining dataset: rclone copy Dropbox:Openedsdata2020 /hdd/datasets/Openedsdata2020 -P
It’s alive!
Conclusion
Hope that my narrative will help someone who is also interested in downloading large folders on a remote server from Dropbox. One of the two ways discussed above may lead you to a desired result. I tried the third way also and it performed very bad, it downloads files very slow and mostly fails to fetch them.
Thanks for reading!
Originally published on Medium in Analytics Vidhya.