General Network Crawling Library
- Morgan Tenney
When working in machine learning, it is important to see where your algorithm makes mistakes. How do you effectively represent the difference between an algorithm's predictions and reality? I will try to find a new and flexible way to visualize these differences so that the user can try to discern patterns.
I will focus specifically on a project of a friend of mine - an analysis of relationship networks and group formation prediction. I think it will be interesting finding a way to represent network predictions and reality in a visually appealing and impactful way.
After reviewing feedback from TA's, I decided to change my focus from visualizing differences in networks to visualizing the crawl, or exploration of an existing network, and the evolution over time of the conclusions one can make. This involves visualizing the accumulation of live data and growth and expansion of the resulting network. The questions I am now trying to solve are:
- How do I visualize the network in an effective way? How do I ensure that the network can grow and continue to be visually compelling?
- How do I convey to the user the reason behind the growth patterns so it doesn't appear aimless?
- How do I give the user freedom? Can they define the data used to form connections/decide where to explore?
Some existing work includes: https://www.youtube.com/watch?v=w9UKUpyqw_4
The author of this project was visualizing different web crawling strategies. While this work is related, it is more specific than what I intend to create.
- Get prototype working. (done)
- Generalize data-fetching. User defines:
- The function to get the data (done)
- The function to process the data so it is fit for display (what are nodes, what are links) (done)
- Generalize priority queue. Allow user to define what they want to explore. (done)
- Make it pretty.
Download the zip: DestinyNET zip