Remove duplicates from multiple lists

Hi,

What is the most effective way to remove duplicates/clones from multiple lists (more than two)? I have x amount of lists which may contain the same vectors. I would like to remove these duplicates from each individual list except from e.g. the first list.

Thanks.

 

You could try using HashSet<T> its unordered data type that allows no duplicates

 

Code (CSharp):

1. public List<object> FirstList;
2. public List<object> SecondList;
3. public List<object> ThirdList;
4.  
5. public void RemoveDuplicates()
6. {  
7.     foreach (object item in FirstList)
8.     {
9.         if (SecondList.Contains(item))
10.         {
11.             SecondList.Remove(item);          
12.         }
13.         if (ThirdList.Contains(item))
14.         {
15.             ThirdList.Remove(item);
16.         }
17.     }
18. }

Edit it to fit your code.

 

Thanks man. Will look into it!

 

Great! Actually I now have the lists in a List<List<Vector2>> but i guess the same could apply to that. Thanks!

 

Although... How would you check for Vector2 duplicates in a List<List<Vector2>>?

 

This is the problem right now:

Code (CSharp):

1. void RemoveDuplicates(List<Lamp> lamps)
2.     {
3.         foreach(Lamp lamp in lamps)
4.         {
5.             foreach(List<Vector2> list in lamp.Hits)
6.             {
7.                 foreach(Vector2 v in list)
8.                 {
9.                     // If previous list contains v, remove v?
10.                 }
11.             }
12.         }
13.     }

lamp.Hits is the List<List<Vector2>>.

 

Hummm I'm not sure, but the first thing that comes to my mind is list.Intersect(otherList).

With that you can get the common objects (duplicated objects) between 2 lists.

 

As a general note be really careful looking for duplicates in vectors. Floats are notoriously hard to compare for equality. And vectors are composed of multiple floats.

Two possible approaches. You can use a for loop and access the previous list with [i-1]. Or you can just store list at the end of each iteration.

 

If comparing floats is tricky, then maybe I should approach the problem differently. See, I have these lamp objects and I cast rays from each lamp (from a lamps script) to 4 points on the individual agents in the scene that are within the 'light cone'. I want the agents to each have a visibility component which is how many of these 4 'bodypoints' are within the light (raycast to from lamp object), but if the agent is within the light cone of multiple lamps how do I then make sure the visibility value does not exceed the number of body points?

Here is what I do now:

Code (CSharp):

1.         List<Lamp> lampList = new List<Lamp>();
2.  
3.         foreach (Transform child in transform)
4.         {
5.             currentChild = child;
6.  
7.             lightRadius = child.GetComponentInChildren<DynamicLight>().lightRadius;
8.             lightAngle = child.GetComponentInChildren<DynamicLight>().RangeAngle;
9.  
10.             Collider2D[] hits = Physics2D.OverlapCircleAll(child.position, lightRadius, perimeterMask);
11.  
12.             Lamp lamp = new Lamp();
13.             lamp.Origin = child.position;
14.  
15.             List<List<Vector2>> lampHits = new List<List<Vector2>>();
16.  
17.             for (int i = 0; i < hits.Length; i++)
18.             {
19.                 if (hits[i] != null)
20.                 {
21.                     Vector2[] boundingPoints = new Vector2[4];
22.  
23.                     boundingPoints[0] = new Vector2(hits[i].bounds.center.x, hits[i].bounds.max.y);
24.                     boundingPoints[1] = new Vector2(hits[i].bounds.min.x, hits[i].bounds.center.y);
25.                     boundingPoints[2] = new Vector2(hits[i].bounds.max.x, hits[i].bounds.center.y);
26.                     boundingPoints[3] = new Vector2(hits[i].bounds.center.x, hits[i].bounds.min.y);
27.  
28.                     foreach (Vector2 bp in boundingPoints)
29.                     {
30.                         if (PointInLight(bp))
31.                         {
32.                             lampHits.Add(CalculateLitPoints(hits[i], boundingPoints));
33.  
34.                             break;
35.                         }
36.                     }
37.                 }
38.             }
39.  
40.             lamp.Hits = lampHits;
41.             lampList.Add(lamp);
42.         }
43.  
44.         // Here I was gonna remove duplicates from the lists so the lamps would not raycast to an object already being raycast to...
45.  
46.         foreach (Lamp lamp in lampList)
47.         {
48.             foreach (List<Vector2> list in lamp.Hits)
49.             {
50.                 foreach(Vector2 v in list)
51.                     Debug.DrawLine(lamp.Origin, v, Color.yellow);
52.             }
53.         }
54.     }

And here is an image og the scene.
http://puu.sh/lBGKs/4640d1d7c8.jpg
Notice that the blue agent has multiple rays cast to the same body points...

Is there a better way?

 

I would consider storing the data in a dictionary with the colliders themselves as keys. Advantages of this approach are you only ever calculate the hit points once per collider. It also means you can check if another lamp has already illuminated a target, and save raycasting against it.

Rather then eliminating duplicate points from the list, we are never creating duplicate points in the first place.

Something like this (heavy) pseudo code. Massage it in, and it should behave as you wish.

Code (CSharp):

1. public class IlluminationHitData {
2.     Vector2[] boundingPoints = new Vector2[4];
3.     bool [] hasBoundingPointBeenHit = new bool[4];
4.  
5.     public IlluminationHitData (Collider2D collider){
6.         boundingPoints[0] = new Vector2(collider.bounds.center.x, collider.bounds.max.y);
7.         boundingPoints[1] = new Vector2(collider.bounds.min.x, collider.bounds.center.y);
8.         boundingPoints[2] = new Vector2(collider.bounds.max.x, collider.bounds.center.y);
9.         boundingPoints[3] = new Vector2(collider.bounds.center.x, collider.bounds.min.y);
10.     }
11. }
12.  
13. public class Illumination : MonoBehaviour {
14.  
15.     void Illuminate (){
16.         Dictionary<Collider2D, IlluminationHitData> illuinationData = new Dictionary<Collider2D, IlluminationHitData>();
17.         foreach (Lamp lamp in yourLampCollection){
18.  
19.             // Do your stuff to find overlaps and colliders
20.          
21.             foreach (Collider2D collider in yourColliderCollection){
22.                 if (!illuminationData.ContainsKey(collider)){
23.                     illuminationData.Add(collider, new IlluminationHitData(collider);
24.                 }
25.  
26.                 IlluminationHitData hitData = illuminationData[collider];
27.              
28.                 for (int i = 0, i < hitData.boundingPoints.length; i++){
29.                     if(!hitData.hasBoundingPointBeenHit[i]){
30.                         Debug.DrawLine (lamp.Origin, hitData.boundingPoints[i], Color.Yellow);
31.                         if (RaycastFromLampHits(hitData.boundingPoints[i]){
32.                             hitData.hasBoundingPointBeenHit[i] = true;
33.                         }
34.                     }
35.                 }
36.             }
37.         }
38.  
39.         foreach (KeyValuePair<Collider2D, IlluminationHitData> dataPoint in illuminationData){
40.             // Do something awesome with the data
41.         }
42.     }
43. }

 

Thanks. That makes total sense. Appreciate it!