In this article, you will learn what you can do to protect your privacy when even the most widespread tools seem to fail this goal.
With increasing levels of online data surveillance, user activity tracking and user profiling, threats to online data security and user privacy continues to worry most of "us", normal users. But, can we do anything to protect our privacy when even the most widespread tools seem to fail this goal? The answer is "yes".
The following video inspired me in trying to contribute:
ProtonMail, Min. 8:50 VIDEO
After some months of hard work, the result is the
Windows Application shown below, a solution which actually works ! ...for no money.
...and to be honest, it works really well ! IP Radar 2 - Intrusion Detection and Prevention in Real Time Based, e.g., on Geographical Locations of Hosts
IP Radar 2 – Main Window
, the abbreviation of " IDPS Intrusion Detection and Prevention System", is generally a sophisticated software tool which most normal users don't have. The closest we use are features offered by some firewalls and anti-viruses, but these tools usually don't detect based on useful criteria like the geographical location of the remote hosts. Therefore, an IDPS is a necessary and meaningful complement, also for home use.
IP Radar 2, a variant of a previous tool I created a long time ago (*), was conceived to fill some gaps between the typical cybersecurity tools available for standard users. The main goal was to have a full open source and lightweight IDPS tool available for anyone.
(*) driven by mere desperation while my computer was being attacked on a daily basis from Russia and other countries, I created a NIDS (Network Intrusion Detection System) which I called IP Radar . That tool actually helped me in discovering attackers and eventually getting rid of them. The Idea in Short
In contrast to the previous "monolithic" tool, which was programmed in Java and used OpenGL, the new "lightweight" tool, programmed in
Python, shall be partially browser-based and make use of the Google Maps API, which is open source and free... or at least that is what I thought.
Very soon, I discovered that since 16
th July 2018, Google Maps offers a price model based on usage. Before starting my research on alternatives, I found out that the variant of the Google Maps API which is offered "for development purposes only" still has enough features to cover the main functionality I had in mind.
The free API is quite choppy, has annoying watermarks, and a very restricted functionality. Nevertheless, after working with it for some months, I got used to it and find it indeed pretty good. Besides, porting the API in future to some other alternative shall be no big deal.
Common to the original IP Radar, I again used
, an excellent free and open-source packet analyzer. In my opinion, this is the de-facto standard tool for every case when it comes to troubleshooting, analysis and development of communication protocols. tshark
To do that, this time, I used
, a python wrapper for tshark, which proved to have a very good performance. pyshark
But the main change from IP Radar to IP Radar 2 is the transition from an IDS to a IDPS, focusing more on "defense" against malicious network activities like Information Theft, Data Breaches, Malicious Ads and Remote Access Tools.
After making my first experience with
while playing around with "machine learning", I realized that the programming language, the de-facto programming language for "artificial intelligence", has a big contributing community with tons of open-source code and libraries, and it has become one of the mainstream programming languages in the world. python
That being said, after a pretty painful learning phase, and after getting used to all specifics of the Python programming language, I discovered its capabilities as a high extensible scripting language.
With increasing complexity and decreasing available time, the need to put things together very quickly and reusing code as much as possible with as less effort as required becomes one of the major targets. Python usually makes clever use of C-code "behind the scenes" in those cases where performance is required while providing a simple and common "front end".
At this point, I have to warn about the fact that the code provided in this article:
was written by a "novice” Python programmer with very scarce time following "intentionally” a very straight-forward way to program and prioritizing simplicity and time over formalities ("extreme programming”)
This iterative method led nevertheless to pretty good results in terms of features and stability, while sticking to a rough architecture.
This explains, for example, many arbitrary or even incorrect function and variable names. Sorry for that.
Nevertheless, I hope you can fade-out the unaesthetic parts and profit from the rest.
I will, for sure, upload a version with improvements sometime. But if someone with the necessary skills and time is willing to create an open source project based on this code, then I would like to be a contributor, just contact me.
While navigating the internet using IP Radar, a tool which showed me the geographical locations on a map of the hosts which my computer connected to, I quickly realized that most of my "normal" web traffic was mainly located in Europe and US. Suspicious or abnormal network activities correlated with hosts located in other countries like Russia, Ukraine, Israel, Malaysia (or was it Thailand?), ..., just to mention some!
Yes, that was quite a ride. My computer was misused as part of a
. European botnet
The fact that I could easily distinguish friend from foe by knowing where the host was located, helped me solve the problem in the end.
The main features of IP Radar 2, a successor of IP Radar, are listed next.
This is one of the main features of IP Radar 2. That is, visualization of the network activity on the world map, using Google Maps. The output file
is generated periodically while capturing, or only once when reading offline from a capture file. map.html
Markers show the geographical locations of hosts, and lines connecting markers represent exchange of network traffic (e.g., TCP or UDP packets).
Different colors and shapes represent different characteristics of network traffic, e.g., for
marker, connection pairs:
red, red: "bad" host – from a location configured as illegal pink, black: "killed" connection to this host orange, orange: "unresolved" host name missing dot in marker, yellow: no answer to "ping" command blue, blue: host in "local" network green, cornflower-blue: "good" host
The height of the marker is proportional to the amount of traffic "sent" to that host. This shall help to easily identify "information leakage".
A thick connection represents a connection which is currently "established".
Arrows show the direction of the traffic.
An information marker in the middle of the Atlantic ocean shows the current number of nodes and other information.
The background of the map can be chosen from among these options:
The main window contains a list of all known nodes (hosts) marked with the corresponding colors. This list provides lots of additional information like number of bytes received from and transmitted to that host.
By clicking on a list-element, the additional information is displayed in a side panel in a more readable form. That information can be copy-pasted.
The main window contains as well many possibilities to "filter" information shown on the map according to different criteria. A change in the filter re-generates
allowing fast and easy focus on specific nodes. map.html
For example, when selecting "Amazon" or "Microsoft" from the combo-box, only those nodes belonging to these companies will be displayed. The combo-box is filled automatically with owners detected during the capture session.
This shall give you a good idea of the amount of "attention" that these companies give to us... or to our data!
The automatic addition of firewall rules to block "bad" hosts can be enabled or disabled any time.
The definition of what a "bad" host is can be set in
, in a white-list or in a black-list containing corresponding names of countries, cities or owners. Other criteria, like bandwidth limit, may be added in future. config.ini Kill
Current "established" connections can be terminated by "killing" the corresponding owner processes on request following one of these criteria:
A specific IP can be selected from the corresponding combo-box which will interactively show hosts having a current active connection.
It is also possible to kill connections automatically. Here are the options:
All killed processes will be shown in a list on the main window:
A combo-box offers all known IPs from the current session in order to select one to ping.
The selected IP can then be pinged on demand by sending an ICMP echo request. The result is updated automatically in the main window and in the map. Hosts which answered the ping request will have a dot in the marker on the map, otherwise the marker will have no dot.
Ping command options:
10 random IPs all known selected IP
A ping can also be sent to "all known” hosts when requested. This process may take some time, that shows why it is convenient to have a separate thread for doing that as we'll see later (
pingResolver.processingThread()). If the ping timeout is configured, e.g., to 250ms and we have 100 hosts which do not answer the request, then it takes 25 seconds to complete this command. But don't worry, this is the reason why we allocate some jobs to different threads. In this tool, it is normal to have some "deferred" results which are displayed later on as soon as available.
A specified number of IP addresses, chosen randomly, may be pinged for test purposes.
The number of random IP addresses to ping can be configured in
Finally, a ping can be sent to each new detected host automatically when setting the corresponding option (checkbox auto):
The sound can be enabled or disabled any time. The tool plays different sounds in different situations like, requesting host resolution, ping request, ping failed, etc. But most importantly, the tool plays an alarm when "bad" hosts are detected or a specified amount of transmitted data is exceeded.
Three files are automatically generated during a capture session (
DATE is a placeholder for the current date and time, e.g., "
report_DATE.csv out_DATE.txt ips_DATE.txt
The first file contains the exact same information seen in the hosts-list on the main window.
The second file contains the output of the application which can be configured to different logging levels.
The third file contains all IP addresses detected. This file or its text can be directly input to
Besides the files listed above, we have the output of tshark in
log_DATE.pcapng, which can be opened any time to restore the session, and map_DATE.html. Storage
In order to optimize for performance, location information of hosts which were resolved in previous sessions is read from
locationsResolved.json during startup and updated with every new location. config.ini
This file allows setting the following parameters:
INTERFACE (if left blank, the first interface will be selected per default)
RUN_AS_ADMIN (set to
True to add rules to the Windows firewall)
SHELL_TO_FILE (to log shell to file or console)
CHECK_PERIOD_IN_SEC (poll cycle of threads – after consuming queues completely)
MY_CITY, MY_COUNTRY, MY_LATITUDE, MY_LONGITUDE,..
USE_WHITE_LIST (True / False)
" AF": " Afghanistan",
" UA": " Ukraine"
" DE": " Germany",
" US": " United States"
WhiteListNotKill=[ (to avoid killing vital processes by accident!)
" BAD GUY",
" The Hacker"
" San Francisco",
" Los Angeles"
] Performance: UI Reaction Time..
For all current features, the tool shows an excellent performance. The user can do lots of things without experiencing any noticeable delays or seeing a "sand-clock"...oh my...I hate sand-clocks! ;-)
The memory used is approximately 200MB.
CHECK_PERIOD_IN_SEC is set to 500ms, the CPU usage is extremely low (<0.1% under steady conditions, with short bursts of up to 10% when peaks of network activity occur which result in detection of many new IPs).
The only thing which some users may need to get used to is the need to "refresh" the
map.html file (by pushing Reload current page in your browser) in order to show the most current information. As a help, the button Show map opens each time a new tab in your default browser. This allows seeing the progress of the session and applying different filters for comparison.
In Firefox, it is also possible to add the
Auto Reload Tab extension. Other browsers may have a similar add on. Status
On the top rightmost side, you find the status changing in form of a "stick" following the sequence of characters "
/", "-", "\", "|" with the period defined by
CHECK_PERIOD_IN_SEC*2.0 (the refresh rate of the main GUI).
The background of the status symbol alternates between gray and green colors.
This shows if the application is running at all, and if if does, if it is running at the right pace.
Loading Files Offline
The tool offers as well the possibility to load packet capture files (
*.pcapng *.pcap *.cap) to analyze them offline. While loading a packet capture file, the main window and the file map.html will be updated in the same manner it happens during the actual capture session.
open file can be used to open capture files offline:
With this simple pattern, repeated five times, we can build the main structure of the application.
Whenever required, we will use a mutex to access common resources in a thread-safe way. Not only the queue itself needs to be protected in this manner, but also subsequent data structures which serve as a temporary repository for data which are consumed from a separate working thread. In Python, we use
for that purpose. Lock
Here, we see a rough overview of the architecture:
After capture of network traffic is started,
tshark will start passing new packets to our
packet_callback(), there we will immediately put them into the
packetQueue. This creates our first "barrier" against the first bottleneck. If
packetQueue is big enough, we'll be able to process the packets unhurriedly. The price for this short breathing pause is the addition of a small delay, which for our purposes is totally negligible.
The game of producing and consuming data will continue over several threads and queues in the pipeline until all information is finally processed. Some tasks are processed in parallel, this allows not blocking or delaying things unnecessarily.
The next thread consuming packets from
processor.processingThread(), which in turn triggers further data processing in 4 subsequent threads.
Those threads will allow "simultaneous" and independent processing of:
ping requests host names resolution killing processes owner of "illegal" network connections adding new blocking rules to the system firewall updating the main window updating the map ( map.html)
processor thread, the results of the other worker threads are retrieved and then stored in
node_dict, which in turn helps updating the main window.
node_dict is a dictionary containing "nodes" of the type
NodeDataClass which has the decorator
@dataclass that allows definition of "fields" with specific types and default values. This was for me an important discovery to help me work in a way that resembles how I work with other programming languages like C, C++ and Java, while getting around the topic
mutable vs. immutable objects in Python.
For orientation, I think it is a good idea to have an overview of the main data structure in this application:
position: int # position in "circle" with center lat, lon
local: bool # local Network? incl. broadcast and multicast
This rather "bulky" class and the corresponding over inflated objects show the simplistic approach followed when designing the application.
Such a massive concentration of information is very handy when it comes to dealing with as few objects as possible. This has the advantage that everything can be kept consistent very easily. For this small tool, this is a good compromise.
Finally, the call to
plotMap() will re-generate the output file
th and final thread is
mainWindow.updateGui(). This thread of course updates our GUI (main window) to show changes in host names, ping responses, killed connections and so on.
The use of 7 threads and 5 queues allows a good workload distribution.
The following diagram shows a typical scenario:
Before you get too confused trying to map this diagram to the code, be aware that this diagram shall only depict the general idea behind the use of multiple threads without holding exactly to the actual code. The general idea shall nevertheless be in both cases the same, namely, don't block and do things in parallel as much as possible.
In the previous diagram, it can be clearly seen that packets A, B, C, D, and E are safely stored in the queue and then wait patiently there until they get processed, without blocking tshark or getting lost.
The following list shows the most important Python scripts:
admin.py badConnectionKiller.py CheckableComboBox.py configuration.py firewallManager.py hostResolver.py IPRadar2.py node.py pingResolver.py processor.py pysharkSniffer.py ui/mainWindow.py
and here a list of other important files:
config.ini gen_exe_with_pyinstaller.bat locationsResolved.json map_DATE.html requirements.txt
For the sake of simplicity, I will only provide a very short description of the most important "building yards" I had myself ongoing for several days or weeks. This shall prevent you from falling into the same pitfalls.
How to Run Python Script with Elevated Privilege on Windows
If you always remember that you have to run the tool as Admin in order to be able to kill some processes or add new rules to the Windows firewall, then you are fine.
If on the other side, you want the application itself to remind you that whenever the option
RUN_AS_ADMIN is set to
config.ini, then you have to follow this link → . Admin Console Commands
Whenever you run out of ideas and you don't want to reinvent the wheel or just don't find the required Python library, then it's time to do things the "old-fashioned way" by executing some console commands from within your Python code.
For that purpose, you shall refer to
With this, you shall be able to execute commands like "
netstat -ano | findstr \"established\" | findstr..." that will allow you checking active TCP connections and the corresponding owner processes, that you may want to kill if required.
Similarly, you can execute "
taskkill", as alternative to
psutil.Process().kill() or "
netsh" to check for existing firewall rules or adding new firewall rules that will block "bad" IPs.
Here, you find a list of the console commands I used so far:
taskkill (alternative to
subprocess.Popen().kill() for killing processes)
netsh advfirewall firewall show rule / add rule
pipreqs (to generate list of Python requirements for
tshark (to obtain available interfaces – feature not provided by pyshark)
Executing external tools is not any different than executing any other console command (as explained above). Therefore,
subprocess.Popen() will help here once again.
I will not reveal how many hours I spent trying to improve the resolution of host names using different methods, different python libraries and workarounds.
Finally, I discovered
. That really saved my life. This command-line utility allows you to easily find all available information about an IP address. That is much better than having only the host name. Whosip
At this point, I have to remark that I've seen sometimes differences between the information provided by
, that's why I added the button " ipnetinfo.exe ipnetinfo” to make a double check if required.
If the option "
all” is selected, this button will call ipnetinfo.exe, passing the file ips_DATE.txt as an input. Otherwise, only the IP address of the selected host is used.
Some "real" Code
Here you find a real piece of code, taken arbitrarily, just as an example for those of you who have never done anything with Python before:
def __pingHost(self, hostIP):
response_list = ping(hostIP, timeout=configuration.PING_TIMEOUT_SEC,
if response_list.rtt_min < configuration.PING_TIMEOUT_SEC:
print( " Ping to IP = ", hostIP)
print( " rtt_min_ms = ", response_list.rtt_min_ms)
print( " rtt_avg_ms = ", response_list.rtt_avg_ms)
if (configuration.PING_TIMEOUT_SEC - response_list.rtt_max) <
configuration.PING_TIMEOUT_SEC* 0. 1:
print( " WARNING! ping response close to max. value, rtt_max_ms = ",
print( " Time out in Ping to IP = ", hostIP)
except Exception as e:
print( " Exception in __pingHost() = ", e)
print( " Exception in __pingHost() to IP = ", hostIP)
I hope this code is self-explanatory and you can grasp what it does and see the main syntax rules in Python.
What is even more important is "why" we need this function. This is explained in the
This is the reason why I focus so much on requirements, architecture, data model, dynamic behavior, lessons learned, etc., and don't mention so much the code itself in this article.
Besides, as an amateur programmer in assembler, BASIC, C, C++, C#, Java, Python, SciLab-script, VHDL, VBA,... I sometimes see the following things as a nightmare:
currAbsPath = currAbsPath.replace(
" \\", " /")
if " b\'" in line:
line = line.replace( " b\'", " ")
self.threadForPacketProcessing = threading.Thread(..)
So I am happy this time to focus only on more abstract topics.
I am nevertheless a passionate programmer, but now and then, I realize that the most important things are actually not the code but what's behind it. I hope you agree to some extent and come along with me in this high-level description of the application.
In parallel, you can download the code to inspect it and run it (you can even generate an
I provided a
READ_ME.txt with the steps you need to follow. IDE, Tools and Libraries
Whenever you start something new and you are not sure how to do it, what tools to use, which alternatives are better for your goal, then you realize that you have to invest quite a lot of time to answer those questions.
Just one example: once I discovered
PyCharm as a great python IDE to do stuff with "machine learning", I soon realized that this tool was not what I need if I want to build IP Radar 2.
: This IDE is just the right thing that I need! Eric6
Free, stable, up to date, with lots of features, combined with PyQt5 to make GUIs, etc.
Nevertheless, for applications "without" a GUI, I would still recommend
. The most important feature I missed in Eric6 is being able to navigate the code jumping to code declarations with one click. This important feature is not supported in Eric6. PyCharm
: This set of cross-platform libraries allow implementing high-level APIs. After a good but hard experience with Qt (using C++) on Ubuntu many years ago, my critics were: PyQt5
heavy environment difficult synch between main application and Designer
I've seen none of that while using Pyqt5 integrated with Eric6.
: This is a python wrapper for tshark. pyshark
: This is a network protocol analyzer. It lets you capture packet data from a live network, or read packets from a previously saved capture file. Note that tshark calls tshark for much of its capture functionality. Note also that you need to install dumpcap . Npcap
: Once you have a working set of packages in your python environment, and you want to be able to install those exact versions elsewhere (otherwise, your tool may not work due to incompatibilities), you may want to use pipreqs. pipreqs
For that, I call
"pipreqs --force ./ --ignore backups" and obtain:
pip freeze only saves the packages that are installed with
pip install in your environment.
pip freeze saves all packages in the environment including those that you don't use in your current project (if you don't have
: This tool lets you install or update packages for python. See also pip install pip.
IP Radar 2, as the name implies, shows on the world map all IP addresses your machine is talking to.
This geographical representation of hosts with different IPs gives the name to this tool, in analogy to a standard Radar used to detect airplanes.
The analogy continues if we consider e.g. that Radars are able to locate flying objects which have a "transponder". This is the usual case in commercial airplanes who identify themselves to "secondary" Radars.
But in case of military air-crafts or smuggling planes, no contact over transponders is possible as they may be turned off intentionally. Then, the secondary Radar is blind.
In those cases, a "primary" Radar is required which locates planes not willing to identify themselves by illuminating a large portion of space with an electromagnetic wave and receiving back the reflected waves from targets within that space.
In IP Radar 2, we resolve the geographical location of hosts and also send a ping request (ICMP echo request) to them to get their replies, being able to calculate round trip time and packet loss statistics.
Some people say that the lack of ping replies does not necessarily mean that the host is willing to hide from us. Some servers try to protect themselves from Denial of Service attacks and do not reply to ping requests, some hosts just won't bother redirecting ICMP requests behind the NAT gateway or a corporate network.
Nevertheless, in my opinion, it is always suspicious when a host does not reply to my pings.
The representation of an airplane on a radar console may show several attributes like height, usually in feet above sea level. Similarly, in IP Radar 2, we show several information like the number of bytes transmitted to a specific host. This is seen in the height of the marker.
The described techniques for intrusion detection and prevention are for sure not the "strongest" against malicious network activity, but they work well
(*), providing a good compromise between complexity, real-time behavior and robustness. Many aspects of the proposed idea can be improved in future.
(*) things that I have detected with this tool so far:
encrypted connections to a Linux mirror in a university in Ecuador – which, of course, I blocked. Probably a malware tunneling stuff? permanent background activities from Amazon, Microsoft and Google. Some of them could be reduced by deactivating unnecessary windows services. But most of it remains a backpack you need to carry when you are a Windows user. drawing tool from Bulgaria without certificate. Which I did not download. annoying user profiling for advertising from companies located in Paris. connections to cheap hosting servers every time my WiFi reconnects.
All these points show clearly that my PC has software running which I don't need and is probably harmful. None of the anti-viruses that I used detected them up to now, but wherever they hide, they are no longer able to do any damage.
Here, we see that the Tool actually helped increasing the security of my PC and is a meaningful complement to antivirus and firewall software.
After a couple of weeks of detecting "bad” guys almost daily, now I can say that I am free. Since 2 months, I only see nice "green” hosts (my firewall has now 85 additional blocking rules!). Why Do We Need Something Like IP Radar 2 ?
Providers of firewalls and antivirus software offer the main measures for network security. Unfortunately, despite all features provided by these tools, our PCs still suffer under massive vulnerability problems. Infections with all kinds of exploits make these tools look like a Swiss cheese.
By combining their capabilities with custom IDPS features, security can be notably improved.
Typical Capture Session / Use Case
So, I wake up early in the morning and while sipping a cup of coffee, and before starting to read the newspaper, I check the world-map on my browser to see if there is something new.
This usually takes only a few seconds. If I only see green bubbles and blue lines, I know for sure that I am safe.
But if I see some red markers and red lines, then I know there may be some trouble.
In that case, a short glance on the map lets me locate the possible intruder in a remote corner of the world where I usually see no activity.
With one click on the corresponding IP address, I get more information displayed that shows that the owner is already on my black-list.
Oh no, a closer look shows that a connection with that IP is established at this very moment!
Damned! The option "
Kill conn.s (auto)" was set to
No problem, on request, I can now kill the process holding that connection by selecting the IP marked in red.
That will solve the problem for now, but what about next time? Oh, now I see that a blocking rule for that IP has been automatically added to my firewall. Everything is fine!
Now I can peacefully start my matutinal reading. ;-)
Using the Code
Besides this article and the links provided in it, I supply together with the code a short
READ_ME.txt file which shall help you with the initial steps. Points of Interest
Borrowing some words from Andy Yen (see link above): "What we have here is just the first step, but it shows that with improving technology privacy doesn't have to be difficult, it doesn't have to be disruptive. Ultimately, privacy depends on each and everyone of us. And we have to protect it now because our online data is more than just a fractions of ones and zeros. It's actually a lot more than that. It's our lives, our personal stories, our friends, our families, and in some ways also our hopes and aspirations. So, now it's the time for us to stand up and say: yes, we do want to live in a world with online privacy. And yes, we can work together to turn this vision into reality!".
For a real-life story about dangers to online-privacy, check these links:
If you want to do even more to protect your privacy check also
AC4QGP (Audio Chat for Quite Good Privacy): Chat using "enhanced" end-to-end-encryption and modulation of audio signal in isolated device, ensuring privacy, anonymity and cybersecurity. History
22 nd May, 2020: Initial version 23 rd February, 2021: Broken links repaired 24 th February, 2021: New code uploaded (new useful feature: node-ID added to find IPs in list easily!) 27 th February, 2021: .zip file with code now includes CPOL license 5 th March, 2021: added link to AC4QGP