url-analyzer/README.md

# URL Analyzer

URL data analyzer and extractor. Detect malicious signs and other useful data associated with URLs.

## About

This program extract various website information based on URL addresses. This data can be used to analyze maliciousness of the given URL.

### Features

**NOTE**: See sample JSON data: [Get file](sample_dataset.json)

To summarize, the program does the following procedures for listed URLs:

- Gets domain registrar
- Gets webpage title and automatically compares it to the domain registrar name
- Gets initial and final destination of a given URL
  - Analyzes whether final destination domain is same than the initial one
- Gets URL redirects and HTTP response status codes
- Fetches WHOIS data
  - Gets domain timestamps such as creation, update and expire days
    - Exact days & days relative to the current day

- Gets content and number of iframes (for detecting possible XSS; Cross-Site Scripting)

- Gets URL references on a webpage
  - **Local** domain referrals
  - **External** URL referrals
  - **Multidot** URLs (ones with `../` in the URL path)
    - Gets domain registrars for each URL

## Requirements

```
Python 3
Python 3 BeautifulSoup4   python-beautifulsoup4
Python 3 whois <= 0.7.3   python-whois; PyPI
Python 3 JSON Schema      python-jsonschema
Python 3 Numpy            python-numpy
Python 3 matplotlib       python-matplotlib
```

**NOTE**: Some Linux distributions may use `python3` executable instead of `python` for Python 3.

### Other requirements

- Jupyter (recommended)
- Working DNS name resolution
- Internet connection

## Code

- `jupyter notebook (python 3)`: [Get file](code/url-analyzer.ipynb)

- `python 3`: [Get file](code/url-analyzer.py)

## Screenshots

The following screenshots are generated with `matplotlib`

### Domains associated with HTML URL data

![](screenshots/domain_figure_hsfi.png)

![](screenshots/domain_figure_tsfi.png)

## Known bugs issues and missing features

- Non-UTF-8 character decoding not implemented

- If multiple JSON data files exist, a wrong JSON data file is likely selected

- Get URLs and other parameters from command line

- More data visualization and compherensive analysis

- Null data may be generated in some cases

- Add (unit) tests

## License

N/A