TensorBoard doesn't show all data points

Question

I was running a very long training (reinforcement learning with 20M steps) and writing summary every 10k steps. In between step 4M and 6M, I saw 2 peaks in my TensorBoard scalar chart for game score, then I let it run and went to sleep. In the morning, it was running at about step 12M, but the peaks between step 4M and 6M that I saw earlier disappeared from the chart. I tried to zoom in and found out that TensorBoard (randomly?) skipped some of the data points. I also tried to export the data but some data point including the peaks are also missing in the exported .csv.

I looked for answers and found this in TensorFlow github page:

TensorBoard uses reservoir sampling to downsample your data so that it can be loaded into RAM. You can modify the number of elements it will keep per tag in tensorboard/backend/server.py.

Has anyone ever modified this server.py file? Where can I find the file and if I installed TensorFlow from source, do I have to recompile it after I modified the file?

Cabbage soup · Accepted Answer · 2018-10-19 05:37:51Z

62

You don't have to change the source code for this, there is a flag called --samples_per_plugin.

Quoting from the help command

--samples_per_plugin: An optional comma separated list of plugin_name=num_samples pairs to explicitly specify how many samples to keep per tag for that plugin. For unspecified plugins, TensorBoard randomly downsamples logged summaries to reasonable values to prevent out-of-memory errors for long running jobs. This flag allows fine control over that downsampling. Note that 0 means keep all samples of that type. For instance, "scalars=500,images=0" keeps 500 scalars and all images. Most users should not need to set this flag. (default: '')

So if you want to have a slider of 100 images, use:

tensorboard --samples_per_plugin images=100

edited Oct 19, 2018 at 5:37

Cabbage soup

1,3941 gold badge18 silver badges27 bronze badges

answered Sep 5, 2018 at 6:16

Phúc Lê

2,6654 gold badges19 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Yesh Over a year ago

This is the real solution.

Qin Heyang Over a year ago

I tried to set --samples_per_plugin scalars=0 and then all my data points are gone. So I set --samples_per_plugin scalars=999999999 and it works perfectly....

Gulzar Over a year ago

Can you please link to the relevant doc?

Sylvain Lesage Over a year ago

The doc is here: github.com/tensorflow/…

paradite Over a year ago

note that changing this for scalar would have an effect on smoothing. I used 0.99 smoothing for default settings, now i need 0.999 smoothing with scalars=3000 for similar-looking graph

dandelion · Accepted Answer · 2017-05-02 17:19:22Z

15

The comment is out of date - it can actually be modified in tensorboard/backend/application.py, in the "Default Size Guidance". By default, it stores 1000 scalars. You can increase that limit arbitrarily, or set it to 0 to store every scalar.

You don't need to recompile TensorBoard, or even download it from source. You could just modify this file in your TensorBoard yourself.

If you install TensorFlow using pip in virtualenv (ubuntu, mac), then within your virtualenv directory the path to application.py should be something like lib/python2.7/site-packages/tensorflow/tensorboard/backend. If you modify that file, you should get the new setting in your tensorboard (when you run tensorboard in that virtualenv). If you're like me, you'll put a print statement too so you can be sure that you're running modified code :)

answered May 2, 2017 at 17:19

dandelion

1,7921 gold badge15 silver badges19 bronze badges

4 Comments

Kerawit Somchaipeng Over a year ago

Thank you so much. I increased the number to 50,000 and it works perfectly. In my case, I'm using anaconda and the file "application.py" is located at "~/.conda/envs/python3/lib/python3.6/site-packages/tensorflow/tensorboard/backend/application.py"

K.Wanter Over a year ago

@dandelion, by default, it stores 1000 scalars, but in my plot, I have 80k steps with 0.5k for each datapoint, yet, 160 points in total, far less than 1000, I suppose this would not be sampled, but displayed all of them, but I see they are still downsampled, what is the machinism here?

Matthew Rahtz Over a year ago

Thanks! For others: specifically, look for scalar_metadata.PLUGIN_NAME.

Vincent Tang Over a year ago

do i have to rerun the training process? cos it still download 1000 rows after I changed the limit (I tried on the saved log file) @MatthewRahtz

Collectives™ on Stack Overflow

TensorBoard doesn't show all data points

2 Answers 2

5 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related