Log ForwardedEvents lost during last analysis period - as reported by Windows Event Tracing (ETW)


Log ForwardedEvents lost during last analysis period - as reported by...

jtangen

We are receiving errors in Internal Logs that state:

Log ForwardedEvents lost 1 with EPS 19 during last analysis period - as reported by Windows Event Tracing (ETW)
Log ForwardedEvents lost 5 with EPS 40 during last analysis period - as reported by Windows Event Tracing (ETW)

These are 2 examples but there are numerous other entries throughout a day.

The collector server(s) specs are:

Two 2.6 GHz processors
8 GB of memory
Version is 19.1.4.0

From https://forum.logbinder.com/Topic79.aspx it appears that is was not a SuperCharger issue but a Windows issue and it was resolved by using a subscription.

We are using subscriptions for the clients to send logs to SuperCharger.

We currently have a minimal amount of machines in each subscription. The most machines in a subscription is around 15.

What are some tasks that can be preformed to troubleshoot this issue?
Is there a simple way to determine the actual source of the entry, for example when it says EPS 19, does that correspond to a machine or subscription?
Tamas Lengyel

jtangen - 4/23/2019
We are receiving errors in Internal Logs that state:

Log ForwardedEvents lost 1 with EPS 19 during last analysis period - as reported by Windows Event Tracing (ETW)
Log ForwardedEvents lost 5 with EPS 40 during last analysis period - as reported by Windows Event Tracing (ETW)

These are 2 examples but there are numerous other entries throughout a day.

The collector server(s) specs are:

Two 2.6 GHz processors
8 GB of memory
Version is 19.1.4.0

From https://forum.logbinder.com/Topic79.aspx it appears that is was not a SuperCharger issue but a Windows issue and it was resolved by using a subscription.

We are using subscriptions for the clients to send logs to SuperCharger.

We currently have a minimal amount of machines in each subscription. The most machines in a subscription is around 15.

What are some tasks that can be preformed to troubleshoot this issue?
Is there a simple way to determine the actual source of the entry, for example when it says EPS 19, does that correspond to a machine or subscription?

EPS stands for Events Per Second.
You can check the Microsoft-Windows-EventCollector/Operational log in Event Viewer for more information. Look for warning event id 501 from EventCollector.
You can refer to articles, such as Avoid Lost Events for troubleshooting.

Tamas Lengyel

Tamas Lengyel - 4/24/2019
jtangen - 4/23/2019
We are receiving errors in Internal Logs that state:

Log ForwardedEvents lost 1 with EPS 19 during last analysis period - as reported by Windows Event Tracing (ETW)
Log ForwardedEvents lost 5 with EPS 40 during last analysis period - as reported by Windows Event Tracing (ETW)

These are 2 examples but there are numerous other entries throughout a day.

The collector server(s) specs are:

Two 2.6 GHz processors
8 GB of memory
Version is 19.1.4.0

From https://forum.logbinder.com/Topic79.aspx it appears that is was not a SuperCharger issue but a Windows issue and it was resolved by using a subscription.

We are using subscriptions for the clients to send logs to SuperCharger.

We currently have a minimal amount of machines in each subscription. The most machines in a subscription is around 15.

What are some tasks that can be preformed to troubleshoot this issue?
Is there a simple way to determine the actual source of the entry, for example when it says EPS 19, does that correspond to a machine or subscription?

EPS stands for Events Per Second.
You can check the Microsoft-Windows-EventCollector/Operational log in Event Viewer for more information. Look for warning event id 501 from EventCollector.
You can refer to articles, such as Avoid Lost Events for troubleshooting.

Over the past few weeks we have done some very in-depth extensive research in to ETW and the “Events Lost” performance counter.  Contrary to widely held understanding, Lost Events does not indicate WEC is failing to deliver events from forwarding computers to the destination event logs.  In fact, our extensive testing has revealed very good news.  Even under heavily overloaded conditions, WEC does not lose events.  In such cases, WEC may slow down and in extreme cases even stop receiving events but events are never lost in a black hole.

This is really good news and means we are re-instrumenting how Supercharger interprets the Lost Events counter.  Our experiments show that this counter still has value for indicating when WEC is overloaded and needs more resources or re-balancing of workload.  And this is important because if WEC is too slow in receiving events or stops accepting events, logs could potentially wrap on source computers before the events are forwarded.  Getting value from the Lost Events counter will take some more research and expect enhancements in the future.  We also plan enhancements for more sophisticated detection of slow or hung WEC collectors in the near future.

But the more immediate action we are taking is changing Supercharger so that it no longer issues the warning you’ve been seeing when it sees an increase in the Lost Events counter because the warning is inaccurate and gives the false impression you are losing events. 

Please update to the latest version 19.7.1 by downloading the package here:  https://www.logbinder.com/Form/SCDownload.  You can perform an in-place upgrade with the downloaded installation package.

We will be publishing a blog soon on the research we did in conjunction with this issue. 

But for now, you can rest easy about this warning.  No events are being lost.  Update to the latest version and you will no longer receive them.

The Supercharger for Windows Event Collection team remains committed to helping your monitor every aspect of health throughout your logging  pipeline.


jtangen

Tamas Lengyel - 7/19/2019
Tamas Lengyel - 4/24/2019
jtangen - 4/23/2019
We are receiving errors in Internal Logs that state:

Log ForwardedEvents lost 1 with EPS 19 during last analysis period - as reported by Windows Event Tracing (ETW)
Log ForwardedEvents lost 5 with EPS 40 during last analysis period - as reported by Windows Event Tracing (ETW)

These are 2 examples but there are numerous other entries throughout a day.

The collector server(s) specs are:

Two 2.6 GHz processors
8 GB of memory
Version is 19.1.4.0

From https://forum.logbinder.com/Topic79.aspx it appears that is was not a SuperCharger issue but a Windows issue and it was resolved by using a subscription.

We are using subscriptions for the clients to send logs to SuperCharger.

We currently have a minimal amount of machines in each subscription. The most machines in a subscription is around 15.

What are some tasks that can be preformed to troubleshoot this issue?
Is there a simple way to determine the actual source of the entry, for example when it says EPS 19, does that correspond to a machine or subscription?

EPS stands for Events Per Second.
You can check the Microsoft-Windows-EventCollector/Operational log in Event Viewer for more information. Look for warning event id 501 from EventCollector.
You can refer to articles, such as Avoid Lost Events for troubleshooting.

Over the past few weeks we have done some very in-depth extensive research in to ETW and the “Events Lost” performance counter.  Contrary to widely held understanding, Lost Events does not indicate WEC is failing to deliver events from forwarding computers to the destination event logs.  In fact, our extensive testing has revealed very good news.  Even under heavily overloaded conditions, WEC does not lose events.  In such cases, WEC may slow down and in extreme cases even stop receiving events but events are never lost in a black hole.

This is really good news and means we are re-instrumenting how Supercharger interprets the Lost Events counter.  Our experiments show that this counter still has value for indicating when WEC is overloaded and needs more resources or re-balancing of workload.  And this is important because if WEC is too slow in receiving events or stops accepting events, logs could potentially wrap on source computers before the events are forwarded.  Getting value from the Lost Events counter will take some more research and expect enhancements in the future.  We also plan enhancements for more sophisticated detection of slow or hung WEC collectors in the near future.

But the more immediate action we are taking is changing Supercharger so that it no longer issues the warning you’ve been seeing when it sees an increase in the Lost Events counter because the warning is inaccurate and gives the false impression you are losing events. 

Please update to the latest version 19.7.1 by downloading the package here:  https://www.logbinder.com/Form/SCDownload.  You can perform an in-place upgrade with the downloaded installation package.

We will be publishing a blog soon on the research we did in conjunction with this issue. 

But for now, you can rest easy about this warning.  No events are being lost.  Update to the latest version and you will no longer receive them.

The Supercharger for Windows Event Collection team remains committed to helping your monitor every aspect of health throughout your logging  pipeline.


We will be installing the updated version of SuperCharger tomorrow.  
GO


Similar Topics


Reading This Topic


Login
Existing Account
Email Address:


Password:


Select a Forum....








LOGbinder Forum


Search