APS Excellence in Physics Education Award
November 2019

Education Prize Logo
Science SPORE Prize
November 2011

NSF Logo
The Open Source Physics Project is supported by NSF DUE-0442581.

Data lost after 10000 frames post and replies

Return to the Data lost after 10000 frames thread
Login to post to this thread

Data lost after 10000 frames
Maik Derksen
2 Posts

Hello,

I use Tracker to track large video files, which seems to work ok.

However, after tracking (roughly) 10000 frames the data dissapears. If I track up to 10050 frames I can export 10050 lines of data, but when I track more than 10050 frames I lose all of the first 10050 lines of data.

Has anyone else experienced this? Is this related perhaps to the problem of not being able to track more than 10000 frames that is mentioned several topics down?

Thank you!


Replies to Data lost after 10000 frames

Re: Data lost after 10000 frames -
lookang Avatar
lookang
245 Posts

yes, if i am not wrong, tracker can currently work well up to 10000 frames.

i did encounter friends who suggest trimming video to below 10000 frames before importing into Tracker for analysis.


OSP@SG blog
OSP@SG Digital Library


Re: Re: Data lost after 10000 frames -
Maik Derksen
2 Posts

Dear Lookang,

Thank you for your reply. It tells me that it is probably not a problem with my settings or PC setup. Trimming the video to clips of 10,000 frames max. would indeed be a workaround.



Re: Re: Re: Data lost after 10000 frames -
Douglas Brown Avatar
Douglas Brown
457 Posts

As I mentioned in the earlier discussion of this 10000 frame issue, the only built-in limit is 300,000 frames (almost 3 hrs at 30fps). So that's not what is causing this.

I wish I had a solution, but I think for now your best bet is to cut your videos into 10000 frame chunks (even smaller is better in terms of speed) and then combining your data in Data Tool or other analysis application.  Doug



Re: Data lost after 10000 frames -
Max Chen
7 Posts

Hmm very interesting. I'm working on a 20,000 frame project right now, but tracker is going slowly. I'll let you know if I have a similar problem around 10,000 frames.

-Max



Re: Re: Data lost after 10000 frames -
Max Chen
7 Posts

Did another 14400 frame video. Tracker performed well finishing the track overnight. Same issue arose however. After refreshing once Tracker was done, data was only available for the last 10,000 frames or so.

Will update as project progresses.



Re: Re: Re: Data lost after 10000 frames -
Max Chen
7 Posts

Updates:

It appears that Tracker consistently cannot retain more than ~10,000 frames worth of autotracker data in a single .trk file. I've been running autotracks on multiple ~22,000 frame .avi files, and in each case I must save a separate .trk file once ~10,000 frames of auto track data is created in order to avoid losing the previous frames. It seems that once tracker hits its 10,000 frame limit, it begins dropping off the oldest frames. For example, if set to track a 15,000 frame .avi, tracker will track to completion but after refreshing and populating the data tables, tracking data is only available for the last 5000-15,000 frames, dropping off the first 5000, or however many frames need to be dropped in order to keep tracker under its apparent 10,000 frame limit.

This appears to be the case regardless of allocated memory usage.

Any fixes would be great, thanks!

Trying to extract motion across a wide band of frequencies from a single time record necessitates long video lengths as well as high frame rates; this results in large total numbers of frames that need to be tracked. Computationally I think my computer could complete a 30,000 frame track in a little under 3 hours, but as it stands right now such a project needs to be closely monitored over the course of the 3 hours in order to extract the data as the 10,000 frame packages are completed.



Re: Re: Re: Re: Data lost after 10000 frames -
Max Chen
7 Posts

Another update:

Tracker successfully tracked and stored autotrack data for frames F0 to F10171.
Later that day did another track and let it run a slight bit too long to F10989, and the autotrack data for the frames from F0 to F4096 became inaccessible, leaving only autotrack data from F4096 to F10989 available.

Weird little bug, seems like a quick fix perhaps? Would greatly appreciate it.



Re: Data lost after 10000 frames -
Douglas Brown Avatar
Douglas Brown
457 Posts

Max, I'm curious to know why you want to mark more than 10000 data points for a single track. I might be able to extend this limit but already at 10000 Tracker is VERY slow and unresponsive. I really think you're better off doing one or both of the following with very long videos:

a. Increase the step size so you skip frames and therefore mark fewer points.
b. Cut the video into shorter segments, track them separately, and combine the data in Data Tool or (if you have more than 10000 data points) a spreadsheet.

Can you tell me why these actions won't work in your situation?  Thanks!



Re: Re: Data lost after 10000 frames -
Max Chen
7 Posts

Unfortunately I cannot increase the step size or reduce the total number of frames. In fact in the long term it would be ideal to be able to track 40,000 or even 50,000 frames. Right now I'm averaging together 10 sets of data of 20,000 frame tracks. It is important to have both high frame rate, and long time records in order to capture the complete spectrum of oscillations that I'm trying to track.

Around 10,000 frames Tracker does become quite slow. Initially I can get Tracker up to about 8 frames per second, but around 10,000 frames this is often approaching ~2 or 3 frames per second.

Thank you very much for your help, Tracker has been an invaluable tool for the initial steps of my data analysis, but I think it might be time to begin tracking using some of MATLAB's built in image processing functions.



Re: Re: Re: Re: Data lost after 10000 frames -
David Timerman
3 Posts

I'll chime in with my work around. The problem is related to viewing/exporting > 10000 data points, not saving them in a .trk file. You can recover your data by viewing chunks of 10 000 data points. After an autotracking session, go to clip settings and set start and end frame to 0 and 9999, and refresh your data window. Then change the settings to 10000 - 19999 and so on. You'll be able to correctly view/export/copy+paste your data. I hope this helps.



Re: Re: Re: Re: Re: Data lost after 10000 frames -
Max Chen
7 Posts

Thank you! This works very well, except it garbles the time data. Not too big of an issue however as the time data can be recreated easily. It seems that the .trk file is holding onto the data, just not displaying it.



OSP Projects:
Open Source Physics - EJS Modeling
Tracker
Physlet Physics
Physlet Quantum Physics
STP Book