All you need to know about Manual Deduplication

All you need to know about Manual Deduplication

In our previous posts we have talked about duplication problem and possible solutions. We have explained how to set up automatic deduplication and hopefully this was helpful. Now we are going to go one step further and explain how to set up manual deduplication and why or when this could be helpful.

If you haven't had time to check out our past post on this subject, you can find them here:

Devices configuration and NetFlow deduplication

Net Admins think about their data and if it is actually correct and deduplicated. This post gives a short explanation of this problem and how to overcome it.

How to solve duplicated NetFlow caused by multiple exporters

This post will help you understand and solve exporters deduplication - what to do when same flow is exported by different devices.

In general, if you have correctly configured exporters (ingress/egress) and decided to enable automatic deduplication by exporting from all devices in flow continuity then all flows in your Traffic Patterns should be automatically deduplicated. However, if this is not the case then it is also possible for you to adjust Traffic Pattern configuration to achieve flow deduplication.

Deduplication based on the central exporter

If you have a central exporter (a netflow exporter through which all desired traffic is passing through) then preventing duplicated Traffic Pattern traffic is easy. You just need to add a filter to the Traffic Pattern in the Exporter section of the Traffic Pattern definition. Add the IP address of the central exporter while include option is set. This will result in Traffic Pattern matching only netflow that was exported by the central exporter.

In our example above, flow that passes and is exported by three routers (R1, R2 and R3) will be taken into account and processed only from central router (R2) since Traffic Pattern includes its IP address in Exporter filter.

Deduplication based on exporters and their interfaces

​If you do not have a central exporter and/or your network topology is more complex, you can prevent duplicated Traffic Patterns by entering exporters and their specific interfaces from which you will either include or exclude traffic, when matching traffic to a Traffic Pattern. That way you can exclude specific interfaces on exporters that would duplicate the traffic.

In the example above, flow travelling via R1 and R2 will not be duplicated since R2 is not an exporter, however flow travelling via R1 and R3 will be duplicated. By excluding Interface Out: Vl3 on Exporter R1 only export from exporter R3 will be processed.

Deduplication based on next hop

In the example below, a flow travelling from Host A to Host B passes via two central routers R1 and R2. As a consequence, one flow is exported and processed to a netflow server twice (by R1 and R2). This should be overcome by adding next hop filter.

The solution is to exclude R2 as Next Hop IP address. This will simply skip all the flows passing from router R1 to R2. Flows will be then matched and processed only by router R2. The same applies for flows from Host B to Host A - excluding R1 as Next Hop will skip flows from R2 to R1.

Deduplication at router interfaces

Alternatively, you can avoid duplicated traffic even on routers themselves. It could be accomplished if you do not configure NetFlow on the interfaces which connect backbone routers.

​With so many solutions, now you can decide whether to use automatic deduplication or one of the manual options. Keep in mind notations that we have given above, so that you can be sure which traffic you are receiving. 

Specific traffic patterns monitoring (Facebook, Yo...
Flow export configuration on Juniper network devic...

Related Posts

By accepting you will be accessing a service provided by a third-party external to


Mailing and Visiting Address:
Soneco d.o.o.
Makenzijeva 24/VI, 11000 Belgrade, Serbia
Phone: +381.11.6356319
Fax: +381.11.2455210 |


linkedin facebook facebook